Oct 30 00:33:39.738928 ip-10-0-139-66 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:33:40.199207 ip-10-0-139-66 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:33:40.199207 ip-10-0-139-66 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:33:40.199207 ip-10-0-139-66 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:33:40.199207 ip-10-0-139-66 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:33:40.202741 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202648 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:40.202741 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202740 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:40.202741 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202743 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:40.202741 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202747 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202750 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202753 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202756 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202759 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202761 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202764 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202767 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202770 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202772 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202775 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202778 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202782 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202785 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202789 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202793 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202796 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202799 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202802 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202809 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:40.202860 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202812 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202815 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202819 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202823 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202826 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202829 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202832 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202835 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202837 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202840 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202842 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202845 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202848 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202851 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202854 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202857 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202860 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202863 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202866 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:40.203315 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202869 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202872 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202874 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202877 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202881 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202884 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202886 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202889 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202897 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202901 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202903 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202906 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202908 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202911 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202914 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202916 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202919 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202921 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202923 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202927 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:40.203790 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202929 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202932 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202935 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202937 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202941 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202944 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202946 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202950 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202952 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202955 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202958 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202961 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202963 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202966 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202969 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202972 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202975 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202978 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202980 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:40.204265 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202983 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202985 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202988 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202990 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202993 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202995 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.202998 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203001 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203003 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203006 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203008 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203011 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203013 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203016 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203018 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203022 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203024 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203027 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203029 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:40.204781 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203032 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203035 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203039 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203042 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203044 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203047 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203939 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203946 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203950 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203956 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203960 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203963 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203965 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203968 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203971 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203974 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203976 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203979 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203981 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:40.205227 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203984 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203987 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203989 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203992 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203995 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.203997 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204000 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204003 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204006 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204009 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204011 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204015 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204017 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204020 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204023 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204025 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204028 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204031 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204033 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204036 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:40.205697 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204038 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204041 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204044 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204047 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204049 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204052 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204054 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204057 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204059 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204061 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204064 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204067 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204071 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204074 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204077 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204080 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204082 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204086 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204089 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204091 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:40.206180 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204094 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204096 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204099 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204102 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204105 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204107 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204110 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204112 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204116 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204120 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204124 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204127 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204130 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204133 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204136 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204139 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204142 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204144 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204147 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:40.206665 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204149 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204152 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204154 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204157 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204159 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204162 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204165 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204167 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204169 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204172 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204175 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204177 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204180 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204182 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204185 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204187 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204191 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204194 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204197 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204199 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:40.207124 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204202 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204204 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204207 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204209 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204212 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204214 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204220 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204223 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204225 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204228 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204231 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204234 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204236 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204240 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204820 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204827 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204830 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204833 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204835 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204838 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:40.207618 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204841 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204843 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204846 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204849 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204852 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204855 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204858 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204861 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204863 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204866 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204869 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204872 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204874 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204876 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204879 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204882 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204885 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204888 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204891 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:40.208100 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204894 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204897 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204899 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204902 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204905 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204907 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204910 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204913 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204915 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204918 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204920 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204923 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204925 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204928 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204931 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204934 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204936 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204939 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204941 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204944 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:40.208631 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204946 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204949 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204951 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204954 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204956 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204960 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204964 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204967 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204970 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204972 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204975 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204978 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204982 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204984 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204986 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204989 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204992 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204995 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.204997 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205000 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:40.209104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205002 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205005 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205008 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205010 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205013 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205016 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205019 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205021 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205024 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205026 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205029 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205031 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205034 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205037 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205039 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205042 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205044 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205047 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205049 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:40.209636 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205052 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205054 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205056 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205059 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205062 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205065 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205067 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205069 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205074 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205078 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205081 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205084 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205086 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205089 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205091 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205094 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205096 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205099 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205102 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:40.210079 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205104 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205107 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205110 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205210 2584 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205218 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205225 2584 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205231 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205235 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205239 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205243 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205248 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205252 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205255 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205259 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205262 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205265 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205269 2584 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205272 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205275 2584 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205278 2584 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205281 2584 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205286 2584 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205289 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205292 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:33:40.210533 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205296 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205300 2584 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205304 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205308 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205311 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205314 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205317 2584 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205321 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205324 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205328 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205330 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205335 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205338 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205341 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205344 2584 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205347 2584 flags.go:64] FLAG: --enable-server="true" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205350 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205355 2584 flags.go:64] FLAG: --event-burst="100" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205358 2584 flags.go:64] FLAG: --event-qps="50" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205361 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205364 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205367 2584 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205373 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205376 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205379 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:33:40.211106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205386 2584 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205389 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205392 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205395 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205398 2584 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205401 2584 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205404 2584 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205407 2584 flags.go:64] FLAG: --feature-gates="" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205412 2584 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205415 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205419 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205424 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205427 2584 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205430 2584 flags.go:64] FLAG: --help="false" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205433 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205436 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205439 2584 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205442 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205446 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205449 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205452 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205455 2584 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205458 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:33:40.211856 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205461 2584 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205464 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205467 2584 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205471 2584 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205474 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205476 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205480 2584 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205482 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205485 2584 flags.go:64] FLAG: --lock-file="" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205490 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205493 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205496 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205502 2584 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205505 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205508 2584 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205511 2584 flags.go:64] FLAG: --logging-format="text" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205514 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205517 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205521 2584 flags.go:64] FLAG: --manifest-url="" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205524 2584 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205530 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205534 2584 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205539 2584 flags.go:64] FLAG: --max-pods="110" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205542 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205545 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:33:40.212430 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205548 2584 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205551 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205554 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205557 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205560 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205579 2584 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205583 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205586 2584 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205590 2584 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205593 2584 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205596 2584 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205599 2584 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205602 2584 flags.go:64] FLAG: --port="10250" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205605 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205608 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c56a0b4b81db9f04" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205611 2584 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205616 2584 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205619 2584 flags.go:64] FLAG: --register-node="true" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205622 2584 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205626 2584 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205629 2584 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205632 2584 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205635 2584 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205640 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205643 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205646 2584 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:33:40.213254 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205650 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205655 2584 flags.go:64] FLAG: --runonce="false" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205658 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205661 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205664 2584 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205667 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205670 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205673 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205676 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205680 2584 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205682 2584 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205685 2584 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205688 2584 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205691 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205694 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205697 2584 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205700 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205706 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205710 2584 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205712 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205717 2584 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205719 2584 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205724 2584 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205727 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205730 2584 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:33:40.213897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205733 2584 flags.go:64] FLAG: --v="2" Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205738 2584 flags.go:64] FLAG: --version="false" Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205742 2584 flags.go:64] FLAG: --vmodule="" Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205747 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.205750 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205888 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205892 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205895 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205901 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205904 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205907 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205910 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205912 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205915 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205918 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205921 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205923 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205926 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205929 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205931 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205934 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:40.214503 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205937 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205939 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205942 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205945 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205949 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205953 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205956 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205959 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205963 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205966 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205969 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205971 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205974 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205977 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205979 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205982 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205984 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205987 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205990 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:40.215018 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205994 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.205997 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206000 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206003 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206005 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206008 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206010 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206013 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206015 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206018 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206021 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206023 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206026 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206028 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206031 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206033 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206036 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206040 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206044 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:40.215470 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206047 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206050 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206054 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206057 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206060 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206062 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206065 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206067 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206070 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206072 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206075 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206077 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206080 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206084 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206087 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206090 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206093 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206095 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206099 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:40.215943 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206102 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206104 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206107 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206109 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206112 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206114 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206117 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206120 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206122 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206124 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206127 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206130 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206132 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206135 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206137 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206141 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206144 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206147 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206149 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206152 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:40.216434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206154 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206157 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206159 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206162 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206164 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206167 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206171 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206173 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206176 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206179 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206182 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206185 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206188 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.206215 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206253 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206257 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:40.216982 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206261 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206263 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206266 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206268 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206271 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206274 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206277 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206279 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206282 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206285 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206288 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206292 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206294 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206297 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206299 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206302 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206305 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206308 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206310 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:40.217392 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206314 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206318 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206321 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206324 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206328 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206331 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206333 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206336 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206339 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206342 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206345 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206347 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206350 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206352 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206355 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206358 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206360 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206362 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206365 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:40.218139 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206368 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206370 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206373 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206375 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206378 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206383 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206385 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206388 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206390 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206393 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206395 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206398 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206400 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206404 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206408 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206411 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206413 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206418 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206420 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:40.218876 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206423 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206426 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206429 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206431 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206434 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206436 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206439 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206442 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206444 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206447 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206450 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206452 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206456 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206459 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206461 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206464 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206466 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206469 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206473 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206476 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:40.219389 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206479 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206481 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206484 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206486 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206489 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206491 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206494 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206497 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206499 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206502 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206506 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206509 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206512 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206515 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206517 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206520 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206523 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206526 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206528 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206531 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:40.219908 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206533 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206536 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206538 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206541 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206543 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206546 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.206550 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.214668 2584 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.214688 2584 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.214704 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.214709 2584 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214777 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214783 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214787 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214790 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214792 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214795 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214798 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214801 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:40.220401 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214803 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214806 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214809 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214812 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214814 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214817 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214820 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214822 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214825 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214827 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214831 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214834 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214836 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214839 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214841 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214844 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214847 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214849 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214852 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214854 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:40.220937 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214857 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214869 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214872 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214875 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214878 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214880 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214883 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214887 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214891 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214894 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214897 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214900 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214903 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214905 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214908 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214910 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214913 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214916 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214918 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:40.221423 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214921 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214923 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214927 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214932 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214935 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214938 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214940 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214943 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214946 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214948 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214951 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214954 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214956 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214959 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214962 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214970 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214973 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214976 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214979 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:40.222038 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214982 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214984 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214987 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214989 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214992 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214995 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.214997 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215000 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215002 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215005 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215007 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215010 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215013 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215015 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215018 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215021 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215023 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215026 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215028 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:40.222652 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215031 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215034 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215037 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215039 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215042 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215044 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215047 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215050 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215053 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215056 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215064 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215067 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215069 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215072 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215075 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215077 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215080 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215083 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215086 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215088 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:40.223104 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215091 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.215122 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215313 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215319 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215329 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215333 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215336 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215339 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215343 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215348 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215351 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215353 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215356 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215359 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215361 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:40.223592 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215364 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215366 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215369 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215371 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215374 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215377 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215380 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215383 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215392 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215395 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215398 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215400 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215403 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215406 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215408 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215411 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215414 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215417 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215419 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215422 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:40.223975 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215424 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215427 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215430 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215433 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215435 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215438 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215440 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215443 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215446 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215448 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215451 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215453 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215456 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215459 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215461 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215463 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215467 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215472 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215475 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:40.224472 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215478 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215481 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215489 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215492 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215496 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215498 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215501 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215503 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215506 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215509 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215511 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215514 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215516 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215519 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215521 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215524 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215527 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215529 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215532 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215534 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:40.225036 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215537 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215539 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215542 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215545 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215548 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215550 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215553 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215555 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215558 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215560 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215563 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215566 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215587 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215590 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215593 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215601 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215604 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215607 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215609 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215612 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:40.225551 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215614 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215617 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215619 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215622 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215624 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215627 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215629 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215632 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215634 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215637 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215639 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215642 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215644 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215647 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.215673 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215711 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:33:40.226052 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215716 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215719 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215722 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215725 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215727 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215730 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215733 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215735 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215738 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215741 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215744 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215746 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215755 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215758 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215760 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215763 2584 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215766 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215768 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215770 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215773 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:33:40.226478 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215776 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215778 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215781 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215783 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215786 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215788 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215791 2584 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215793 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215796 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215798 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215801 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215803 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215806 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215808 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215811 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215813 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215816 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215818 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215821 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215823 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:33:40.227035 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215826 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215829 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215831 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215834 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215837 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215848 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215850 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215853 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215856 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215859 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215861 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215864 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215867 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215870 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215874 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215877 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215880 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215882 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215885 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:33:40.227833 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215888 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215890 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215892 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215895 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215898 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215900 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215903 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215905 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215907 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215910 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215913 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215915 2584 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215918 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215921 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215924 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215926 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215929 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215931 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215935 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215945 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:33:40.228434 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215948 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215951 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215954 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215957 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215960 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215962 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215965 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215967 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215970 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215973 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215975 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215978 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215981 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215983 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215986 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215988 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215991 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215995 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.215998 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:33:40.229067 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.216002 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.216005 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.216007 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.216010 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.216013 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.216016 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:40.216019 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.216090 2584 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.223710 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.224760 2584 server.go:1014] "Starting client certificate rotation" Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.224881 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.224938 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.227223 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:33:40.229824 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.228719 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:33:40.243831 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.243810 2584 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:33:40.247968 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.247949 2584 log.go:25] "Validated CRI v1 image API" Oct 30 00:33:40.248863 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.248833 2584 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:33:40.252648 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.252626 2584 fs.go:135] Filesystem UUIDs: map[1f7dada4-297a-4354-9bba-95ad61b13ec7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7e5ce436-68db-4639-a77d-5cae854f295e:/dev/nvme0n1p3] Oct 30 00:33:40.252713 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.252647 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 00:33:40.258111 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.257997 2584 manager.go:217] Machine: {Timestamp:2026-10-30 00:33:40.257485672 +0000 UTC m=+0.384625507 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102254 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2036d8dc443e6c95e4d788c053e09d SystemUUID:ec2036d8-dc44-3e6c-95e4-d788c053e09d BootID:edd3eea2-a8f3-4166-bf58-4d1ae73ea320 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:e7:d9:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:e7:d9:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:eb:23:b9:7e:29 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:33:40.258111 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.258105 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:33:40.258216 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.258187 2584 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:33:40.262060 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.262028 2584 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:33:40.262262 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.262062 2584 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:33:40.262311 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.262275 2584 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:33:40.262311 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.262284 2584 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:33:40.262311 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.262300 2584 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:33:40.262387 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.262319 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:33:40.263559 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.263547 2584 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:33:40.268322 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.268309 2584 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:33:40.268483 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.268473 2584 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:33:40.269337 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.269327 2584 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:33:40.269373 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.269345 2584 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:33:40.269373 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.269357 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:33:40.269373 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.269366 2584 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:33:40.269492 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.269375 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:33:40.269940 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.269924 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:33:40.275431 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.275402 2584 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:33:40.282626 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.282609 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:33:40.283655 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.283642 2584 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:33:40.283709 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.283673 2584 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:33:40.283843 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.283831 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:33:40.283879 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.283848 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:33:40.283879 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.283855 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:33:40.283879 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.283860 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:33:40.283879 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.283865 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:33:40.283879 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.283872 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:33:40.284582 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.284559 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:33:40.284631 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.284588 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:33:40.284631 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.284604 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:33:40.284631 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.284611 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:33:40.284631 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.284620 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:33:40.284631 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.284629 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:33:40.285497 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.285478 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:33:40.285497 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.285493 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:33:40.289522 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.289504 2584 server.go:1263] "Started kubelet" Oct 30 00:33:40.289651 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.289595 2584 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:33:40.289700 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.289617 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:33:40.289700 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.289691 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:33:40.290405 ip-10-0-139-66 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:33:40.291868 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.291795 2584 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:33:40.293002 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.292981 2584 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:33:40.297387 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.297358 2584 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 00:33:40.299207 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.299192 2584 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:33:40.299269 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.299242 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:33:40.300086 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.299630 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:40.300242 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.300225 2584 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:33:40.300242 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.300234 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:33:40.300365 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.300245 2584 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:33:40.300510 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.300494 2584 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:33:40.300593 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.300512 2584 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:33:40.301689 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.301668 2584 factory.go:153] Registering CRI-O factory Oct 30 00:33:40.301689 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.301688 2584 factory.go:223] Registration of the crio container factory successfully Oct 30 00:33:40.301878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.301765 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:33:40.301878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.301773 2584 factory.go:55] Registering systemd factory Oct 30 00:33:40.301878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.301779 2584 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:33:40.301878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.301792 2584 factory.go:103] Registering Raw factory Oct 30 00:33:40.302264 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.302248 2584 manager.go:1199] Started watching for new ooms in manager Oct 30 00:33:40.303491 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.303472 2584 manager.go:319] Starting recovery of all containers Oct 30 00:33:40.313114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.313092 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:33:40.315173 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.315153 2584 manager.go:324] Recovery completed Oct 30 00:33:40.319708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.319691 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:40.321450 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.321429 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:40.321543 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.321467 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:40.321543 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.321486 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:40.322171 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.322154 2584 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:33:40.322171 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.322167 2584 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:33:40.322255 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.322184 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:40.322255 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.322190 2584 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:33:40.323003 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.322982 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:40.323086 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.323014 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:40.323086 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.323024 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:40.324710 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.324697 2584 policy_none.go:50] "Start" Oct 30 00:33:40.324761 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.324719 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:33:40.324761 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.324730 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:33:40.326397 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.326379 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 00:33:40.326730 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.326719 2584 policy_none.go:44] "Start" Oct 30 00:33:40.345590 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.345554 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lg5mm" Oct 30 00:33:40.348789 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.348755 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 00:33:40.348789 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.348766 2584 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 00:33:40.348956 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.348897 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 00:33:40.350270 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.349051 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18e3276bbbceefaa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-10-30 00:33:40.289474474 +0000 UTC m=+0.416614307,LastTimestamp:2026-10-30 00:33:40.289474474 +0000 UTC m=+0.416614307,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.361053 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lg5mm" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.361455 2584 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.366385 2584 manager.go:341] "Starting Device Plugin manager" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.366605 2584 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.366620 2584 server.go:87] "Starting device plugin registration server" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.366887 2584 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.366900 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.367000 2584 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.367080 2584 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.367088 2584 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.367635 2584 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.367685 2584 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.373603 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18e3276bbdb6d972 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-10-30 00:33:40.321450354 +0000 UTC m=+0.448590190,LastTimestamp:2026-10-30 00:33:40.321450354 +0000 UTC m=+0.448590190,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.373668 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.375309 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.376556 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.376592 2584 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.376616 2584 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:33:40.379658 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.376678 2584 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 00:33:40.393470 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.393447 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:40.467958 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.467877 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:40.468885 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.468870 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:40.468972 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.468898 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:40.468972 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.468908 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:40.468972 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.468936 2584 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.477141 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.477120 2584 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal"] Oct 30 00:33:40.477192 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.477184 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:40.478169 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.478154 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:40.478239 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.478182 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:40.478239 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.478195 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:40.480532 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.480519 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:40.480666 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.480649 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.480731 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.480688 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:40.481442 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.481423 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:40.481527 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.481455 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:40.481527 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.481465 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:40.481527 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.481488 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:40.481527 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.481499 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:40.481527 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.481469 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:40.483650 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.483634 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.483742 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.483674 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:40.484382 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.484363 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:40.484435 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.484390 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:40.484435 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.484400 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:40.487836 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.487817 2584 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.487932 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.487841 2584 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-66.ec2.internal\": node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:40.501169 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.501141 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.501169 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.501172 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.501350 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.501190 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07ffc31fbc22d538689cf3a7345b3440-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" (UID: \"07ffc31fbc22d538689cf3a7345b3440\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.510436 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.510413 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-66.ec2.internal\" not found" node="ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.513101 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.513080 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-66.ec2.internal\" not found" node="ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.523243 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.523218 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:40.602232 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.602193 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.602232 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.602228 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.602446 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.602248 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07ffc31fbc22d538689cf3a7345b3440-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" (UID: \"07ffc31fbc22d538689cf3a7345b3440\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.602446 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.602298 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07ffc31fbc22d538689cf3a7345b3440-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" (UID: \"07ffc31fbc22d538689cf3a7345b3440\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.602446 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.602312 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.602446 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.602312 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.623749 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.623715 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:40.724672 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.724602 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:40.815577 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.815538 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.818872 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:40.818850 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:33:40.824951 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.824933 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:40.926554 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:40.926496 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.027067 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:41.026973 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.127171 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:41.127130 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.225187 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.225140 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 00:33:41.225711 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.225321 2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:33:41.227430 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:41.227407 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.327566 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:41.327471 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.367020 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.366933 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:28:40 +0000 UTC" deadline="2028-04-02 06:27:02.33482027 +0000 UTC" Oct 30 00:33:41.367020 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.367009 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12485h53m20.967815539s" Oct 30 00:33:41.407417 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.407380 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:33:41.408755 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.408730 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:33:41.408865 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.408775 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:33:41.408865 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.408790 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:33:41.427701 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:41.427665 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.492023 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:41.491986 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ffc31fbc22d538689cf3a7345b3440.slice/crio-9845a4b8e5d215caa91fcc36fb85ede18b33a1e62172f98a640366efdd8f314c WatchSource:0}: Error finding container 9845a4b8e5d215caa91fcc36fb85ede18b33a1e62172f98a640366efdd8f314c: Status 404 returned error can't find the container with id 9845a4b8e5d215caa91fcc36fb85ede18b33a1e62172f98a640366efdd8f314c Oct 30 00:33:41.492598 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:41.492565 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49830c06cd5e40b61be21088e0ddfb5.slice/crio-ee4611656cca54a51dfb522d4effab40913e518515f83b1ecca6f6a2eb2e1e74 WatchSource:0}: Error finding container ee4611656cca54a51dfb522d4effab40913e518515f83b1ecca6f6a2eb2e1e74: Status 404 returned error can't find the container with id ee4611656cca54a51dfb522d4effab40913e518515f83b1ecca6f6a2eb2e1e74 Oct 30 00:33:41.497859 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.497837 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:33:41.528805 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:41.528764 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.623887 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.623795 2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:41.629196 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:41.629174 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.723775 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.723736 2584 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:41.730168 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:41.730141 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.831303 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:41.831260 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:33:41.894340 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.894255 2584 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:41.900320 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.900287 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:33:41.930833 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.930799 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:33:41.932194 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.932158 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:33:41.954861 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:41.954826 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:33:42.272261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.272157 2584 apiserver.go:52] "Watching apiserver" Oct 30 00:33:42.301604 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.301557 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 00:33:42.307189 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.307157 2584 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:33:42.307646 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.307620 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-wzp7p","openshift-network-operator/iptables-alerter-s8jbb","openshift-ovn-kubernetes/ovnkube-node-fjgxn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal","openshift-multus/network-metrics-daemon-8blbw","openshift-dns/node-resolver-wdz28","kube-system/global-pull-secret-syncer-2rkgf","kube-system/konnectivity-agent-xvxrc","openshift-cluster-node-tuning-operator/tuned-rhvzs","openshift-image-registry/node-ca-274sk","openshift-multus/multus-additional-cni-plugins-6485j","kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal","openshift-multus/multus-t6nzm"] Oct 30 00:33:42.310771 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.310736 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.312858 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.312837 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:42.312950 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.312912 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:33:42.315712 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.315684 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.317927 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.317905 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.320501 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.320474 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:42.320601 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.320548 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:33:42.323015 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.322996 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:42.323123 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.323076 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:33:42.325771 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.325747 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:42.327990 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.327967 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.328967 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.328948 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:33:42.329798 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.329775 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk5pr\"" Oct 30 00:33:42.329895 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.329883 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:33:42.332457 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.331967 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.334841 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.334821 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.334947 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.334851 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:33:42.337426 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.337402 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.341474 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.340025 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.346820 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.346734 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:33:42.370866 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.370642 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:33:42.370866 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.370705 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x46qh" Oct 30 00:33:42.383904 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.383815 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" event={"ID":"07ffc31fbc22d538689cf3a7345b3440","Type":"ContainerStarted","Data":"9845a4b8e5d215caa91fcc36fb85ede18b33a1e62172f98a640366efdd8f314c"} Oct 30 00:33:42.385600 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.385564 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" event={"ID":"b49830c06cd5e40b61be21088e0ddfb5","Type":"ContainerStarted","Data":"ee4611656cca54a51dfb522d4effab40913e518515f83b1ecca6f6a2eb2e1e74"} Oct 30 00:33:42.402234 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.402205 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:33:42.411624 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.411529 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:42.411624 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.411564 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-socket-dir-parent\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.411624 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.411598 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-k8s-cni-cncf-io\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.411624 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.411626 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.411986 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.411652 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:42.411986 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.411704 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2685357-831c-41ea-bc51-f86b156f7eb9-host-slash\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.411986 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.411777 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-kubelet\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.411986 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.411827 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-systemd-units\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.411986 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.411862 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.412281 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412001 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-konnectivity-ca\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:42.412281 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412083 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-kubelet\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.412281 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412115 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-tuned\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.412281 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412162 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljxqv\" (UniqueName: \"kubernetes.io/projected/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-kube-api-access-ljxqv\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.412281 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412208 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-script-lib\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.412281 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412234 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-registration-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.412281 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412275 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5dbm\" (UniqueName: \"kubernetes.io/projected/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kube-api-access-d5dbm\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412323 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-lib-modules\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412339 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412356 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412380 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-bin\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412411 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412438 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-sys-fs\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412484 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-multus\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412576 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-multus-certs\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412604 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfcwt\" (UniqueName: \"kubernetes.io/projected/7e97bcaa-c31d-45fb-913d-9f9516905bdc-kube-api-access-dfcwt\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412650 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-kubernetes\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412666 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-sys\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412681 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-host\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.412708 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412697 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-kubelet-config\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412719 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd212437-e8f3-4212-befe-c217498b0923-ovn-node-metrics-cert\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412738 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-var-lib-kubelet\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412759 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cnibin\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412798 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd9c3e-2c40-45f6-b808-76dde38075ef-tmp-dir\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412824 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-dbus\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412856 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-slash\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412887 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cni-binary-copy\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.412904 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-modprobe-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413042 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fbd9c3e-2c40-45f6-b808-76dde38075ef-hosts-file\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413077 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2q99\" (UniqueName: \"kubernetes.io/projected/d2685357-831c-41ea-bc51-f86b156f7eb9-kube-api-access-d2q99\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413117 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-system-cni-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413142 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2685357-831c-41ea-bc51-f86b156f7eb9-iptables-alerter-script\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413163 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-host\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413200 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-ovn\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413287 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-config\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.413333 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413305 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysconfig\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413328 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-log-socket\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413346 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-etc-selinux\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413377 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54j2g\" (UniqueName: \"kubernetes.io/projected/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-kube-api-access-54j2g\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413423 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-conf\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413463 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413491 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-etc-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413515 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413534 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cnibin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413549 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-daemon-config\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413739 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-serviceca\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413772 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-netns\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413824 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413864 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-systemd\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413898 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413930 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-device-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.414176 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.413967 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-agent-certs\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414041 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-hostroot\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414082 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjt6f\" (UniqueName: \"kubernetes.io/projected/cae5e6bb-b286-46da-81b9-be3d7ae18631-kube-api-access-rjt6f\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414131 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbn7\" (UniqueName: \"kubernetes.io/projected/8fbd9c3e-2c40-45f6-b808-76dde38075ef-kube-api-access-tmbn7\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414167 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbtr9\" (UniqueName: \"kubernetes.io/projected/38d225e8-d97d-482d-9b91-33d2473d4915-kube-api-access-fbtr9\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414224 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-bin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414258 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414285 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-systemd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414312 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-var-lib-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414405 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-node-log\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414434 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-env-overrides\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414460 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gft\" (UniqueName: \"kubernetes.io/projected/fd212437-e8f3-4212-befe-c217498b0923-kube-api-access-q5gft\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414486 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-socket-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414519 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-system-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414549 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-netns\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414574 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-run\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414608 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.415054 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414725 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-netd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.415693 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414752 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-conf-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.415693 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414777 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-etc-kubernetes\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.415693 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.414805 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-tmp\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.438129 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.438095 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:33:42.438316 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.438279 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:33:42.438491 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.438472 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:33:42.438580 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.438543 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:33:42.438674 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.438583 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:33:42.438674 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.438661 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:33:42.438785 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.438729 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x59g5\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439012 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439019 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439071 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439086 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439117 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qpxdw\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439129 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439119 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439114 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439230 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439388 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439435 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qff2g\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439493 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s9968\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439482 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439532 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439560 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439583 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439593 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:33:42.439611 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439612 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:33:42.440491 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439692 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:33:42.440491 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439735 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b4svw\"" Oct 30 00:33:42.440491 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439755 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2pzq\"" Oct 30 00:33:42.440491 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.439776 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tcw96\"" Oct 30 00:33:42.447326 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.447299 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:33:42.447448 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.447375 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ltmbf\"" Oct 30 00:33:42.447494 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.447480 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:33:42.515179 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515142 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-var-lib-kubelet\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.515376 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515188 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cnibin\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.515376 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515217 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd9c3e-2c40-45f6-b808-76dde38075ef-tmp-dir\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.515376 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515234 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-dbus\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:42.515376 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515253 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-slash\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.515376 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515276 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cni-binary-copy\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.515376 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515278 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-var-lib-kubelet\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.515376 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515299 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-modprobe-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.515376 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515318 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cnibin\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.515376 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515347 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fbd9c3e-2c40-45f6-b808-76dde38075ef-hosts-file\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515383 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2q99\" (UniqueName: \"kubernetes.io/projected/d2685357-831c-41ea-bc51-f86b156f7eb9-kube-api-access-d2q99\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515415 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-system-cni-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515419 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-modprobe-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515443 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2685357-831c-41ea-bc51-f86b156f7eb9-iptables-alerter-script\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515471 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-host\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515506 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-ovn\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515532 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-config\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515550 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysconfig\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515565 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-log-socket\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515581 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-etc-selinux\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515606 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-dbus\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515598 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54j2g\" (UniqueName: \"kubernetes.io/projected/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-kube-api-access-54j2g\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515633 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-conf\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515648 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd9c3e-2c40-45f6-b808-76dde38075ef-tmp-dir\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515702 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fbd9c3e-2c40-45f6-b808-76dde38075ef-hosts-file\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.515878 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515659 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-host\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515387 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-slash\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516085 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cni-binary-copy\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516145 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-config\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516161 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-system-cni-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.515651 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516189 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-ovn\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516211 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-etc-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516227 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516241 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516224 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysconfig\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516265 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-log-socket\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516275 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cnibin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516307 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-daemon-config\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516313 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516334 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-etc-selinux\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516336 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-serviceca\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.516667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516365 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-netns\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516392 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516395 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-etc-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516417 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-systemd\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516441 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516462 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-device-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516486 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-agent-certs\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516512 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-hostroot\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516522 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-conf\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516537 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjt6f\" (UniqueName: \"kubernetes.io/projected/cae5e6bb-b286-46da-81b9-be3d7ae18631-kube-api-access-rjt6f\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516548 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516565 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbn7\" (UniqueName: \"kubernetes.io/projected/8fbd9c3e-2c40-45f6-b808-76dde38075ef-kube-api-access-tmbn7\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516579 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-netns\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516365 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cnibin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516591 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbtr9\" (UniqueName: \"kubernetes.io/projected/38d225e8-d97d-482d-9b91-33d2473d4915-kube-api-access-fbtr9\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516618 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-bin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516645 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.517545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516707 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-systemd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516735 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-var-lib-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516733 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2685357-831c-41ea-bc51-f86b156f7eb9-iptables-alerter-script\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516761 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-node-log\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516785 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-env-overrides\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516813 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gft\" (UniqueName: \"kubernetes.io/projected/fd212437-e8f3-4212-befe-c217498b0923-kube-api-access-q5gft\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516851 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-systemd\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516853 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-socket-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516896 2584 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516953 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-socket-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516953 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-system-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.516993 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-serviceca\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517001 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-netns\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517019 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-system-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517028 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-run\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517084 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-systemd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517148 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517182 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.518396 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517203 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-netd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517218 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-conf-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517237 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-etc-kubernetes\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517253 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-tmp\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517271 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517287 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-socket-dir-parent\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517303 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-k8s-cni-cncf-io\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517321 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517400 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517419 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2685357-831c-41ea-bc51-f86b156f7eb9-host-slash\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517437 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-kubelet\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517457 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-systemd-units\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517475 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517493 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-konnectivity-ca\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517513 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-kubelet\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517539 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-tuned\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517556 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxqv\" (UniqueName: \"kubernetes.io/projected/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-kube-api-access-ljxqv\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.519261 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517577 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-script-lib\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517578 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-daemon-config\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517598 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-registration-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517618 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dbm\" (UniqueName: \"kubernetes.io/projected/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kube-api-access-d5dbm\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517660 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-lib-modules\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517680 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-var-lib-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517689 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517693 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517813 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517812 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.517877 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-bin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.518374 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.518463 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-socket-dir-parent\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.518497 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-k8s-cni-cncf-io\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.518783 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.518852 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:43.018828821 +0000 UTC m=+3.145968668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.518901 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2685357-831c-41ea-bc51-f86b156f7eb9-host-slash\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.520114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.518942 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-kubelet\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.518945 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.518976 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-systemd-units\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519164 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-lib-modules\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519197 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-etc-kubernetes\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519204 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-conf-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519262 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-netd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519300 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-node-log\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519338 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519353 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-netns\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519367 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-bin\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519394 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-kubelet\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519412 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519450 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-sys-fs\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519483 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-multus\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519515 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-multus-certs\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.519363 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519560 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-device-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.521112 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519603 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfcwt\" (UniqueName: \"kubernetes.io/projected/7e97bcaa-c31d-45fb-913d-9f9516905bdc-kube-api-access-dfcwt\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519626 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-kubernetes\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519667 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-sys\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519694 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-host\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519719 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-kubelet-config\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519732 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-run\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519756 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd212437-e8f3-4212-befe-c217498b0923-ovn-node-metrics-cert\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519778 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-hostroot\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519809 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519859 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-host\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519878 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-kubelet-config\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519907 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-bin\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519923 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-multus\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519969 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-registration-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.519979 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-sys-fs\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.520009 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-konnectivity-ca\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.520009 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-multus-certs\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.520046 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-sys\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.522008 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.520093 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-kubernetes\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.522946 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.520165 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret podName:5ad24271-c6fb-4ac0-b2b5-b915db5b61ec nodeName:}" failed. No retries permitted until 2026-10-30 00:33:43.020143331 +0000 UTC m=+3.147283178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret") pod "global-pull-secret-syncer-2rkgf" (UID: "5ad24271-c6fb-4ac0-b2b5-b915db5b61ec") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:42.522946 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.520741 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-script-lib\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.522946 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.520970 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-agent-certs\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:42.522946 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.521311 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-env-overrides\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.528728 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.528702 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd212437-e8f3-4212-befe-c217498b0923-ovn-node-metrics-cert\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.529020 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.529000 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-tmp\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.530660 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.530614 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-tuned\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.542748 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.542711 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:42.542748 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.542743 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:42.542748 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.542757 2584 projected.go:196] Error preparing data for projected volume kube-api-access-544sc for pod openshift-network-diagnostics/network-check-target-wzp7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:42.543008 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:42.542837 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc podName:67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:43.04281651 +0000 UTC m=+3.169956360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-544sc" (UniqueName: "kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc") pod "network-check-target-wzp7p" (UID: "67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:42.544079 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.544050 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2q99\" (UniqueName: \"kubernetes.io/projected/d2685357-831c-41ea-bc51-f86b156f7eb9-kube-api-access-d2q99\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.545303 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.545280 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54j2g\" (UniqueName: \"kubernetes.io/projected/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-kube-api-access-54j2g\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.545400 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.545278 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbn7\" (UniqueName: \"kubernetes.io/projected/8fbd9c3e-2c40-45f6-b808-76dde38075ef-kube-api-access-tmbn7\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.549057 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.549030 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjt6f\" (UniqueName: \"kubernetes.io/projected/cae5e6bb-b286-46da-81b9-be3d7ae18631-kube-api-access-rjt6f\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.549260 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.549236 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbtr9\" (UniqueName: \"kubernetes.io/projected/38d225e8-d97d-482d-9b91-33d2473d4915-kube-api-access-fbtr9\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:42.549643 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.549622 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxqv\" (UniqueName: \"kubernetes.io/projected/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-kube-api-access-ljxqv\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.555424 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.555395 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gft\" (UniqueName: \"kubernetes.io/projected/fd212437-e8f3-4212-befe-c217498b0923-kube-api-access-q5gft\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.555424 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.555420 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfcwt\" (UniqueName: \"kubernetes.io/projected/7e97bcaa-c31d-45fb-913d-9f9516905bdc-kube-api-access-dfcwt\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.555589 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.555512 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dbm\" (UniqueName: \"kubernetes.io/projected/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kube-api-access-d5dbm\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.631299 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.631257 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdz28" Oct 30 00:33:42.644632 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:42.644600 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbd9c3e_2c40_45f6_b808_76dde38075ef.slice/crio-0d92191b1e485cfcff938ff26c2b3ead98855d8109dbf0604a32368ab8468df5 WatchSource:0}: Error finding container 0d92191b1e485cfcff938ff26c2b3ead98855d8109dbf0604a32368ab8468df5: Status 404 returned error can't find the container with id 0d92191b1e485cfcff938ff26c2b3ead98855d8109dbf0604a32368ab8468df5 Oct 30 00:33:42.665822 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.665788 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:42.670411 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.670375 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:33:42.670545 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.670501 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:33:42.672562 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.672507 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t6nzm" Oct 30 00:33:42.678225 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:42.678191 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286e9e4c_1a89_4c7c_90b8_222e5f48af2c.slice/crio-6abc4c3ed00b4ee32479a627d9d7430fc6d11552e76c7f5f95ca2497d49c12ca WatchSource:0}: Error finding container 6abc4c3ed00b4ee32479a627d9d7430fc6d11552e76c7f5f95ca2497d49c12ca: Status 404 returned error can't find the container with id 6abc4c3ed00b4ee32479a627d9d7430fc6d11552e76c7f5f95ca2497d49c12ca Oct 30 00:33:42.679471 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.679262 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:33:42.686605 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.686353 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:33:42.686831 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:42.686794 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb15cf1a_dd0d_41d8_b34e_e25029799f9f.slice/crio-51bc6520f9691e99fd90ab64a62afa1968b716593bc45b53a2552f954b2fe70d WatchSource:0}: Error finding container 51bc6520f9691e99fd90ab64a62afa1968b716593bc45b53a2552f954b2fe70d: Status 404 returned error can't find the container with id 51bc6520f9691e99fd90ab64a62afa1968b716593bc45b53a2552f954b2fe70d Oct 30 00:33:42.692059 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:42.692030 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b32a95_e7a4_4ba3_8735_5fb574e470ee.slice/crio-684da8a76cee02366fb360422152c8e343f2aee1b0af13755d7c15dd805dbb8e WatchSource:0}: Error finding container 684da8a76cee02366fb360422152c8e343f2aee1b0af13755d7c15dd805dbb8e: Status 404 returned error can't find the container with id 684da8a76cee02366fb360422152c8e343f2aee1b0af13755d7c15dd805dbb8e Oct 30 00:33:42.693208 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.693186 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-274sk" Oct 30 00:33:42.697385 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:42.697362 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2685357_831c_41ea_bc51_f86b156f7eb9.slice/crio-6432c349df88d876c232e16662063d2336fb299a8d991324afc3580c506fd8cc WatchSource:0}: Error finding container 6432c349df88d876c232e16662063d2336fb299a8d991324afc3580c506fd8cc: Status 404 returned error can't find the container with id 6432c349df88d876c232e16662063d2336fb299a8d991324afc3580c506fd8cc Oct 30 00:33:42.701539 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:42.701517 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae5e6bb_b286_46da_81b9_be3d7ae18631.slice/crio-66925b3529cd4b22202f6ea66c8ffe975357a9678375fdd3a56cd30071c605a2 WatchSource:0}: Error finding container 66925b3529cd4b22202f6ea66c8ffe975357a9678375fdd3a56cd30071c605a2: Status 404 returned error can't find the container with id 66925b3529cd4b22202f6ea66c8ffe975357a9678375fdd3a56cd30071c605a2 Oct 30 00:33:42.708843 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:42.708811 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17afbfde_0e2f_4411_bdb2_bc9b3f5a04f3.slice/crio-ffc949e9eb352b9cd88bef4636bb659979fc4e5bcc5441338ebaa2ee2ebfacca WatchSource:0}: Error finding container ffc949e9eb352b9cd88bef4636bb659979fc4e5bcc5441338ebaa2ee2ebfacca: Status 404 returned error can't find the container with id ffc949e9eb352b9cd88bef4636bb659979fc4e5bcc5441338ebaa2ee2ebfacca Oct 30 00:33:42.715944 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:42.715915 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:33:42.730155 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:33:42.730121 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e97bcaa_c31d_45fb_913d_9f9516905bdc.slice/crio-fb56063e9dc94da2af53ceb99cb3f0689a8b27e30c8a33d7f84144b78a6d219a WatchSource:0}: Error finding container fb56063e9dc94da2af53ceb99cb3f0689a8b27e30c8a33d7f84144b78a6d219a: Status 404 returned error can't find the container with id fb56063e9dc94da2af53ceb99cb3f0689a8b27e30c8a33d7f84144b78a6d219a Oct 30 00:33:43.022505 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.022464 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:43.022719 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.022533 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:43.022795 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:43.022735 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:43.022795 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:43.022790 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret podName:5ad24271-c6fb-4ac0-b2b5-b915db5b61ec nodeName:}" failed. No retries permitted until 2026-10-30 00:33:44.022775273 +0000 UTC m=+4.149915094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret") pod "global-pull-secret-syncer-2rkgf" (UID: "5ad24271-c6fb-4ac0-b2b5-b915db5b61ec") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:43.023196 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:43.023170 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:43.023315 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:43.023231 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:44.023215014 +0000 UTC m=+4.150354851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:43.123083 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.122954 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:43.123167 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:43.123150 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:43.123213 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:43.123172 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:43.123213 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:43.123189 2584 projected.go:196] Error preparing data for projected volume kube-api-access-544sc for pod openshift-network-diagnostics/network-check-target-wzp7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:43.123283 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:43.123242 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc podName:67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:44.123224079 +0000 UTC m=+4.250363914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-544sc" (UniqueName: "kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc") pod "network-check-target-wzp7p" (UID: "67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:43.374216 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.374048 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:28:42 +0000 UTC" deadline="2028-08-09 04:39:52.78074926 +0000 UTC" Oct 30 00:33:43.374216 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.374109 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15580h6m9.406644597s" Oct 30 00:33:43.394441 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.394341 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" event={"ID":"07ffc31fbc22d538689cf3a7345b3440","Type":"ContainerStarted","Data":"fce96918bbeb5c3f32c4953e16cc3e9c417eba8500ddb1e26addefe0f727d273"} Oct 30 00:33:43.397440 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.397352 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-274sk" event={"ID":"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3","Type":"ContainerStarted","Data":"ffc949e9eb352b9cd88bef4636bb659979fc4e5bcc5441338ebaa2ee2ebfacca"} Oct 30 00:33:43.400207 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.400097 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s8jbb" event={"ID":"d2685357-831c-41ea-bc51-f86b156f7eb9","Type":"ContainerStarted","Data":"6432c349df88d876c232e16662063d2336fb299a8d991324afc3580c506fd8cc"} Oct 30 00:33:43.402534 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.402451 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t6nzm" event={"ID":"c7b32a95-e7a4-4ba3-8735-5fb574e470ee","Type":"ContainerStarted","Data":"684da8a76cee02366fb360422152c8e343f2aee1b0af13755d7c15dd805dbb8e"} Oct 30 00:33:43.406458 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.406370 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" event={"ID":"fd212437-e8f3-4212-befe-c217498b0923","Type":"ContainerStarted","Data":"60bdad1d49af2da468593a41842778cfb68f915e02f3a9d9687da2cfbca923a8"} Oct 30 00:33:43.409628 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.409544 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdz28" event={"ID":"8fbd9c3e-2c40-45f6-b808-76dde38075ef","Type":"ContainerStarted","Data":"0d92191b1e485cfcff938ff26c2b3ead98855d8109dbf0604a32368ab8468df5"} Oct 30 00:33:43.412670 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.411378 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerStarted","Data":"fb56063e9dc94da2af53ceb99cb3f0689a8b27e30c8a33d7f84144b78a6d219a"} Oct 30 00:33:43.416886 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.416859 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" event={"ID":"cae5e6bb-b286-46da-81b9-be3d7ae18631","Type":"ContainerStarted","Data":"66925b3529cd4b22202f6ea66c8ffe975357a9678375fdd3a56cd30071c605a2"} Oct 30 00:33:43.427973 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.427940 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" event={"ID":"fb15cf1a-dd0d-41d8-b34e-e25029799f9f","Type":"ContainerStarted","Data":"51bc6520f9691e99fd90ab64a62afa1968b716593bc45b53a2552f954b2fe70d"} Oct 30 00:33:43.429981 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.429872 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvxrc" event={"ID":"286e9e4c-1a89-4c7c-90b8-222e5f48af2c","Type":"ContainerStarted","Data":"6abc4c3ed00b4ee32479a627d9d7430fc6d11552e76c7f5f95ca2497d49c12ca"} Oct 30 00:33:43.441745 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:43.441680 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" podStartSLOduration=1.441661697 podStartE2EDuration="1.441661697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:43.441178837 +0000 UTC m=+3.568318682" watchObservedRunningTime="2026-10-30 00:33:43.441661697 +0000 UTC m=+3.568801547" Oct 30 00:33:44.032314 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.032187 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:44.032314 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.032298 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:44.032566 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.032451 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:44.032566 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.032535 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:46.032502253 +0000 UTC m=+6.159642091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:44.033032 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.033010 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:44.033122 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.033074 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret podName:5ad24271-c6fb-4ac0-b2b5-b915db5b61ec nodeName:}" failed. No retries permitted until 2026-10-30 00:33:46.033057632 +0000 UTC m=+6.160197457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret") pod "global-pull-secret-syncer-2rkgf" (UID: "5ad24271-c6fb-4ac0-b2b5-b915db5b61ec") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:44.133224 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.133049 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:44.133408 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.133263 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:44.133408 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.133296 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:44.133408 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.133312 2584 projected.go:196] Error preparing data for projected volume kube-api-access-544sc for pod openshift-network-diagnostics/network-check-target-wzp7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:44.133408 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.133381 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc podName:67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:46.133358574 +0000 UTC m=+6.260498421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-544sc" (UniqueName: "kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc") pod "network-check-target-wzp7p" (UID: "67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:44.377273 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.375842 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:28:42 +0000 UTC" deadline="2028-07-02 11:06:39.604460852 +0000 UTC" Oct 30 00:33:44.377273 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.375888 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14674h32m55.228577337s" Oct 30 00:33:44.377831 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.377340 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:44.377831 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.377508 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:33:44.377831 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.377707 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:44.377988 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.377863 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:33:44.377988 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.377966 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:44.378084 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:44.378049 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:33:44.449504 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.448168 2584 generic.go:356] "Generic (PLEG): container finished" podID="b49830c06cd5e40b61be21088e0ddfb5" containerID="f0ac33b7f52ce594dfea950a4c93120cf4c006d1b9d49ce757e379b7bf091c91" exitCode=0 Oct 30 00:33:44.449504 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:44.449261 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" event={"ID":"b49830c06cd5e40b61be21088e0ddfb5","Type":"ContainerDied","Data":"f0ac33b7f52ce594dfea950a4c93120cf4c006d1b9d49ce757e379b7bf091c91"} Oct 30 00:33:45.458842 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:45.458091 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" event={"ID":"b49830c06cd5e40b61be21088e0ddfb5","Type":"ContainerStarted","Data":"7c543ba374efefa9f41f25210438dd2985c3c80839da47c273ee2b34bfa26ef0"} Oct 30 00:33:45.505775 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:45.505601 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" podStartSLOduration=3.505582902 podStartE2EDuration="3.505582902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:45.504324898 +0000 UTC m=+5.631464743" watchObservedRunningTime="2026-10-30 00:33:45.505582902 +0000 UTC m=+5.632722750" Oct 30 00:33:46.051255 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:46.051163 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:46.051255 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:46.051227 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:46.051436 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.051377 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:46.051491 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.051443 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret podName:5ad24271-c6fb-4ac0-b2b5-b915db5b61ec nodeName:}" failed. No retries permitted until 2026-10-30 00:33:50.05142499 +0000 UTC m=+10.178564827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret") pod "global-pull-secret-syncer-2rkgf" (UID: "5ad24271-c6fb-4ac0-b2b5-b915db5b61ec") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:46.051533 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.051513 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:46.051566 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.051551 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:50.051538199 +0000 UTC m=+10.178678024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:46.152060 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:46.152015 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:46.152246 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.152201 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:46.152246 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.152226 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:46.152246 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.152240 2584 projected.go:196] Error preparing data for projected volume kube-api-access-544sc for pod openshift-network-diagnostics/network-check-target-wzp7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:46.152501 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.152302 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc podName:67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:50.152284356 +0000 UTC m=+10.279424202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-544sc" (UniqueName: "kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc") pod "network-check-target-wzp7p" (UID: "67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:46.377268 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:46.377183 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:46.377445 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.377306 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:33:46.379969 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:46.379941 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:46.380130 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.380076 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:33:46.380194 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:46.380161 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:46.380257 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:46.380235 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:33:46.463505 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:46.463464 2584 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="f82f5e2962d3ed8ed7615a5d52d65b1d94a28cefc8bd418a3f27c3783b8eb177" exitCode=0 Oct 30 00:33:46.464542 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:46.464490 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerDied","Data":"f82f5e2962d3ed8ed7615a5d52d65b1d94a28cefc8bd418a3f27c3783b8eb177"} Oct 30 00:33:47.475203 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:47.475156 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvxrc" event={"ID":"286e9e4c-1a89-4c7c-90b8-222e5f48af2c","Type":"ContainerStarted","Data":"8c7a66aa96c2b5bd8248ad4d814639ecc765b907cc0f80b258efc29a657af8a7"} Oct 30 00:33:48.380043 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:48.378970 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:48.380043 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:48.379126 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:33:48.380043 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:48.379575 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:48.380043 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:48.379715 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:33:48.380043 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:48.379796 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:48.380043 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:48.379885 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:33:48.482282 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:48.482215 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-274sk" event={"ID":"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3","Type":"ContainerStarted","Data":"d2a93c5820b2463915cc811d46b2fbac31f5a969d653101c59b7677c53bcc625"} Oct 30 00:33:48.529644 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:48.529539 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xvxrc" podStartSLOduration=3.370659451 podStartE2EDuration="7.529515928s" totalImagesPullingTime="4.158856477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:47.523915609 +0000 UTC m=+7.651055471" watchObservedRunningTime="2026-10-30 00:33:48.529515928 +0000 UTC m=+8.656655777" Oct 30 00:33:49.486468 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:49.486366 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" event={"ID":"fb15cf1a-dd0d-41d8-b34e-e25029799f9f","Type":"ContainerStarted","Data":"0599dedf271b1cd3c10d1815f193a9668786d6859d9f07378384200f35370b41"} Oct 30 00:33:50.085081 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:50.085039 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:50.085289 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:50.085100 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:50.085289 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.085243 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:50.085423 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.085313 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret podName:5ad24271-c6fb-4ac0-b2b5-b915db5b61ec nodeName:}" failed. No retries permitted until 2026-10-30 00:33:58.085294851 +0000 UTC m=+18.212434677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret") pod "global-pull-secret-syncer-2rkgf" (UID: "5ad24271-c6fb-4ac0-b2b5-b915db5b61ec") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:50.085715 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.085686 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:50.085812 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.085751 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:58.085734186 +0000 UTC m=+18.212874005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:50.185767 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:50.185720 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:50.185939 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.185850 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:50.185939 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.185876 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:50.185939 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.185890 2584 projected.go:196] Error preparing data for projected volume kube-api-access-544sc for pod openshift-network-diagnostics/network-check-target-wzp7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:50.186095 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.185951 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc podName:67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:58.18593173 +0000 UTC m=+18.313071567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-544sc" (UniqueName: "kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc") pod "network-check-target-wzp7p" (UID: "67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:50.382326 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:50.380593 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:50.382326 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.380718 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:33:50.382326 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:50.381101 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:50.382326 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.381193 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:33:50.382326 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:50.382146 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:50.382326 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.382226 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:33:50.396196 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:50.396160 2584 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb15cf1a_dd0d_41d8_b34e_e25029799f9f.slice/crio-8fdb7fc96ee0e76744265d349a15385aafd9d4f80a1b90dffa314000cba26877.scope\": RecentStats: unable to find data in memory cache]" Oct 30 00:33:50.429767 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:50.429711 2584 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:33:50.493384 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:50.493290 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" event={"ID":"fb15cf1a-dd0d-41d8-b34e-e25029799f9f","Type":"ContainerStarted","Data":"8fdb7fc96ee0e76744265d349a15385aafd9d4f80a1b90dffa314000cba26877"} Oct 30 00:33:51.375419 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.375297 2584 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:33:50.429730485Z","UUID":"e050253a-449d-4bf9-948a-51e303f6e016","Handler":null,"Name":"","Endpoint":""} Oct 30 00:33:51.377218 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.377173 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:33:51.377218 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.377222 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:33:51.497235 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.497199 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdz28" event={"ID":"8fbd9c3e-2c40-45f6-b808-76dde38075ef","Type":"ContainerStarted","Data":"497a817f5eb39ced52f6fac6c371f32c8b6ee9e6088d758e5a21d568faab6a3b"} Oct 30 00:33:51.500144 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.500113 2584 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="63e7207ac23b170ab2e432103f6c222c995f3eb8c6e579f98abe66a906827ed4" exitCode=0 Oct 30 00:33:51.500290 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.500197 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerDied","Data":"63e7207ac23b170ab2e432103f6c222c995f3eb8c6e579f98abe66a906827ed4"} Oct 30 00:33:51.502911 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.502694 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t6nzm" event={"ID":"c7b32a95-e7a4-4ba3-8735-5fb574e470ee","Type":"ContainerStarted","Data":"7976904e3ba61eb9fcc20446b8803a884c4226aea73fdd9d29b7d33a820d2320"} Oct 30 00:33:51.540506 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.540463 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-274sk" podStartSLOduration=5.280557508 podStartE2EDuration="10.540447034s" totalImagesPullingTime="5.259889526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:48.530068519 +0000 UTC m=+8.657208367" watchObservedRunningTime="2026-10-30 00:33:51.540447034 +0000 UTC m=+11.667586873" Oct 30 00:33:51.541303 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.541233 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wdz28" podStartSLOduration=2.4800316000000002 podStartE2EDuration="10.541220445s" totalImagesPullingTime="8.061188845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:51.540707498 +0000 UTC m=+11.667847336" watchObservedRunningTime="2026-10-30 00:33:51.541220445 +0000 UTC m=+11.668360287" Oct 30 00:33:51.641369 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.641300 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-t6nzm" podStartSLOduration=2.442540626 podStartE2EDuration="10.641286718s" totalImagesPullingTime="8.198746092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:51.587508186 +0000 UTC m=+11.714648028" watchObservedRunningTime="2026-10-30 00:33:51.641286718 +0000 UTC m=+11.768426560" Oct 30 00:33:51.645567 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.645536 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:51.646479 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:51.646457 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:52.377785 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:52.377753 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:52.377785 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:52.377780 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:52.378032 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:52.377939 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:33:52.378315 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:52.378292 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:52.378417 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:52.378403 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:33:52.378505 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:52.378482 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:33:52.506938 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:52.506899 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s8jbb" event={"ID":"d2685357-831c-41ea-bc51-f86b156f7eb9","Type":"ContainerStarted","Data":"668c097a493cda4dd51a2b2c0e4f39bb5358754284d696becd71f6076e114108"} Oct 30 00:33:52.510114 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:52.510086 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerStarted","Data":"ffa74123a4520de509180c8dc63c626b442a61903e7dfc0839dcad7bfb743c18"} Oct 30 00:33:52.513590 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:52.513549 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" event={"ID":"fb15cf1a-dd0d-41d8-b34e-e25029799f9f","Type":"ContainerStarted","Data":"ed6dc88d2eae41ee7feb656f8f59ef8b8cab9d20abe7078fd9b6254539d7816a"} Oct 30 00:33:52.549004 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:52.548863 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s8jbb" podStartSLOduration=3.372683169 podStartE2EDuration="11.548844093s" totalImagesPullingTime="8.176160924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:52.548075633 +0000 UTC m=+12.675215476" watchObservedRunningTime="2026-10-30 00:33:52.548844093 +0000 UTC m=+12.675983939" Oct 30 00:33:52.652608 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:52.652523 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" podStartSLOduration=3.194132222 podStartE2EDuration="11.652502126s" totalImagesPullingTime="8.458369904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:52.652126567 +0000 UTC m=+12.779266411" watchObservedRunningTime="2026-10-30 00:33:52.652502126 +0000 UTC m=+12.779641972" Oct 30 00:33:53.517215 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:53.517172 2584 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="ffa74123a4520de509180c8dc63c626b442a61903e7dfc0839dcad7bfb743c18" exitCode=0 Oct 30 00:33:53.517884 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:53.517245 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerDied","Data":"ffa74123a4520de509180c8dc63c626b442a61903e7dfc0839dcad7bfb743c18"} Oct 30 00:33:53.519400 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:53.519382 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" event={"ID":"cae5e6bb-b286-46da-81b9-be3d7ae18631","Type":"ContainerStarted","Data":"9457306a693788c222402241f21bbdc26784e1d036e4bd4d500216819ed46e90"} Oct 30 00:33:53.626619 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:53.626500 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" podStartSLOduration=1.96262074 podStartE2EDuration="12.626484558s" totalImagesPullingTime="10.663863818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:33:53.625745212 +0000 UTC m=+13.752885059" watchObservedRunningTime="2026-10-30 00:33:53.626484558 +0000 UTC m=+13.753624400" Oct 30 00:33:54.377583 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:54.377528 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:54.377754 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:54.377528 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:54.377820 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:54.377795 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:33:54.377897 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:54.377670 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:33:54.377897 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:54.377883 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:54.377982 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:54.377966 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:33:55.526855 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:55.526819 2584 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="6e030d222943f2c37714c9288572740eb794dff5077defc303e549f8e970c0cf" exitCode=0 Oct 30 00:33:55.527512 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:55.526891 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerDied","Data":"6e030d222943f2c37714c9288572740eb794dff5077defc303e549f8e970c0cf"} Oct 30 00:33:56.032541 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:56.032502 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:56.032806 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:56.032674 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:33:56.033297 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:56.033277 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:33:56.377604 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:56.377310 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:56.377604 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:56.377310 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:56.377604 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:56.377449 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:33:56.377604 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:56.377458 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:56.377604 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:56.377550 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:33:56.377947 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:56.377665 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:33:57.534478 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:57.534444 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerStarted","Data":"aca48198ddbe248903fe5479359678f1f5198c3cc0e62bbdf56d59b8b7915ddb"} Oct 30 00:33:58.146858 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.146682 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:58.147309 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.146871 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:58.147309 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.146826 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:58.147309 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.146960 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:58.147309 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.146993 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.146973858 +0000 UTC m=+34.274113692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:33:58.147309 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.147015 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret podName:5ad24271-c6fb-4ac0-b2b5-b915db5b61ec nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.147005692 +0000 UTC m=+34.274145512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret") pod "global-pull-secret-syncer-2rkgf" (UID: "5ad24271-c6fb-4ac0-b2b5-b915db5b61ec") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:33:58.247357 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.247314 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:58.247539 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.247482 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:33:58.247539 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.247506 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:33:58.247539 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.247516 2584 projected.go:196] Error preparing data for projected volume kube-api-access-544sc for pod openshift-network-diagnostics/network-check-target-wzp7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:58.247681 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.247566 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc podName:67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.247552778 +0000 UTC m=+34.374692611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-544sc" (UniqueName: "kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc") pod "network-check-target-wzp7p" (UID: "67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:33:58.377074 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.376993 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:33:58.377074 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.377058 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:33:58.377231 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.377064 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:33:58.377231 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.377156 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:33:58.377299 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.377272 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:33:58.377376 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:33:58.377358 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:33:58.539416 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.539379 2584 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="aca48198ddbe248903fe5479359678f1f5198c3cc0e62bbdf56d59b8b7915ddb" exitCode=0 Oct 30 00:33:58.539889 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.539461 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerDied","Data":"aca48198ddbe248903fe5479359678f1f5198c3cc0e62bbdf56d59b8b7915ddb"} Oct 30 00:33:58.542652 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.542632 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" event={"ID":"fd212437-e8f3-4212-befe-c217498b0923","Type":"ContainerStarted","Data":"af5c251b7eb20f0b5e1eda4eacb00ac28bee6a6873be81826e940d56fa148d02"} Oct 30 00:33:58.542719 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.542692 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" event={"ID":"fd212437-e8f3-4212-befe-c217498b0923","Type":"ContainerStarted","Data":"96e8f3a6938dc64e7e0c5a9958d2a5ba2ade51dfc84a3ba509704730161d3fb1"} Oct 30 00:33:58.542754 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.542725 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" event={"ID":"fd212437-e8f3-4212-befe-c217498b0923","Type":"ContainerStarted","Data":"fe733cb5d3bd79230e951e7434924c48094705b5445b3a1daeb675d37e7d8f25"} Oct 30 00:33:58.542787 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.542754 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" event={"ID":"fd212437-e8f3-4212-befe-c217498b0923","Type":"ContainerStarted","Data":"c0679c8af2955dcc80b22679f24e6ebddbcf33ac3f92aa6f685653fe6a5b1288"} Oct 30 00:33:58.542787 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.542782 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" event={"ID":"fd212437-e8f3-4212-befe-c217498b0923","Type":"ContainerStarted","Data":"2cc1ebb49787fe1d652652e2476d75e86bc60f3daa37d6ec15709306120363df"} Oct 30 00:33:58.542843 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:58.542810 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" event={"ID":"fd212437-e8f3-4212-befe-c217498b0923","Type":"ContainerStarted","Data":"9213a18863019e1c0a26028dac453a6994401249b248d5d124f3d598e06d8c40"} Oct 30 00:33:59.548426 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:59.548392 2584 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="4261a9b702d6dda2215725bb24f97ea15250a36f68d6d2ad67a1c7aa6ba1a85d" exitCode=0 Oct 30 00:33:59.548922 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:33:59.548437 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerDied","Data":"4261a9b702d6dda2215725bb24f97ea15250a36f68d6d2ad67a1c7aa6ba1a85d"} Oct 30 00:34:00.380197 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:00.380118 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:00.380345 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:00.380220 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:00.380345 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:00.380235 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:34:00.380345 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:00.380317 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:34:00.380503 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:00.380370 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:00.380503 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:00.380431 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:34:00.553418 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:00.553379 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" event={"ID":"fd212437-e8f3-4212-befe-c217498b0923","Type":"ContainerStarted","Data":"435312dc36afcc53a0adcc1afb11992898e6fdb877ba696050092b66c93ecb55"} Oct 30 00:34:00.556459 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:00.556431 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6485j" event={"ID":"7e97bcaa-c31d-45fb-913d-9f9516905bdc","Type":"ContainerStarted","Data":"26186b90d08325a0ea126d9b591435478cac22ac7ecb824b75b59b7073ddd662"} Oct 30 00:34:00.635234 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:00.635122 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6485j" podStartSLOduration=8.76140737 podStartE2EDuration="19.635106079s" totalImagesPullingTime="10.873698709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:00.620481767 +0000 UTC m=+20.747621610" watchObservedRunningTime="2026-10-30 00:34:00.635106079 +0000 UTC m=+20.762245920" Oct 30 00:34:02.377645 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:02.377452 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:02.378290 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:02.377472 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:02.378290 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:02.377736 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:34:02.378290 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:02.377513 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:02.378290 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:02.377816 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:34:02.378290 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:02.377865 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:34:02.564449 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:02.564353 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" event={"ID":"fd212437-e8f3-4212-befe-c217498b0923","Type":"ContainerStarted","Data":"e63023f5859d96be025a0410782da76a2b6b67617005329a1b326567fe65a625"} Oct 30 00:34:02.564650 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:02.564619 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:02.564710 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:02.564690 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:02.580603 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:02.580559 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:02.580758 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:02.580681 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:02.664445 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:02.664383 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" podStartSLOduration=6.576134491 podStartE2EDuration="21.664370447s" totalImagesPullingTime="15.088235956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:02.64972035 +0000 UTC m=+22.776860192" watchObservedRunningTime="2026-10-30 00:34:02.664370447 +0000 UTC m=+22.791510288" Oct 30 00:34:03.567071 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:03.567038 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:34:04.356397 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:04.356363 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wzp7p"] Oct 30 00:34:04.356676 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:04.356503 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:04.356676 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:04.356642 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:34:04.359238 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:04.359203 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rkgf"] Oct 30 00:34:04.359358 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:04.359319 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:04.359452 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:04.359423 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:34:04.371482 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:04.371446 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8blbw"] Oct 30 00:34:04.371890 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:04.371871 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:04.372161 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:04.372139 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:34:04.569447 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:04.569419 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:34:06.377624 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:06.377593 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:06.378101 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:06.377593 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:06.378101 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:06.377724 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:34:06.378101 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:06.377787 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:34:06.378101 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:06.377605 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:06.378101 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:06.377871 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:34:06.712913 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:06.712837 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:08.377753 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:08.377719 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:08.377753 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:08.377752 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:08.378227 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:08.377723 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:08.378227 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:08.377848 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:34:08.378227 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:08.377929 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:34:08.378227 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:08.378019 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:34:10.377825 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.377794 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:10.378234 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:10.377881 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzp7p" podUID="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" Oct 30 00:34:10.378234 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.377938 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:10.378234 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.377994 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:10.378234 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:10.378078 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rkgf" podUID="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" Oct 30 00:34:10.378234 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:10.378162 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8blbw" podUID="38d225e8-d97d-482d-9b91-33d2473d4915" Oct 30 00:34:10.705722 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.705416 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeReady" Oct 30 00:34:10.705851 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.705841 2584 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 00:34:10.755218 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.755181 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kkhpn"] Oct 30 00:34:10.762156 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.762129 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xkfl8"] Oct 30 00:34:10.762307 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.762257 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" Oct 30 00:34:10.765101 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.765082 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w2hfm"] Oct 30 00:34:10.765202 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.765191 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" Oct 30 00:34:10.768019 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.767997 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6d46m"] Oct 30 00:34:10.768143 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.768102 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" Oct 30 00:34:10.770390 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.770369 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77dbcdb46-8kvk8"] Oct 30 00:34:10.770531 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.770514 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:10.774257 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.774239 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gbrdh"] Oct 30 00:34:10.776081 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.776058 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.780274 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.780248 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-696c77d7b7-f5pjl"] Oct 30 00:34:10.780467 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.780405 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.784222 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.784204 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tvh9g"] Oct 30 00:34:10.784355 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.784336 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:10.787067 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.787049 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b6dd9b44c-l6dl6"] Oct 30 00:34:10.787194 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.787176 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:10.793420 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.793224 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 00:34:10.793420 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.793393 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bpt2p\"" Oct 30 00:34:10.794432 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.793522 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 00:34:10.794432 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.793647 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.794432 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.793729 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 00:34:10.794560 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.794467 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj"] Oct 30 00:34:10.794631 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.794613 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:10.794720 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.793339 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.799265 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.797111 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 00:34:10.799265 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.797220 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9b4g8\"" Oct 30 00:34:10.799618 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.799597 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 00:34:10.800976 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.800916 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79"] Oct 30 00:34:10.801957 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.801931 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" Oct 30 00:34:10.808794 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.808761 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb"] Oct 30 00:34:10.808945 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.808898 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:10.812607 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.812556 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj"] Oct 30 00:34:10.812769 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.812682 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:10.818158 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.818133 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc"] Oct 30 00:34:10.819095 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.819076 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:10.823260 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.823240 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl"] Oct 30 00:34:10.823380 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.823366 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:10.826031 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.826008 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gbrwr"] Oct 30 00:34:10.827378 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.827360 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" Oct 30 00:34:10.829737 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.829709 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kcpgq"] Oct 30 00:34:10.831738 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.831719 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:10.834669 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.834620 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.834960 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.834852 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.835181 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.835156 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8brp\" (UniqueName: \"kubernetes.io/projected/a2a1f298-60ee-4efe-8cec-88db4036a4e8-kube-api-access-s8brp\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.835320 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.835305 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.835407 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.835340 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.835407 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.835383 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2a1f298-60ee-4efe-8cec-88db4036a4e8-serving-cert\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.835508 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.835447 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.835906 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.835884 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36441a12-4446-4269-b6fd-61d82a00d242-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:10.835997 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.835924 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:10.835997 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.835964 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.836092 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.835997 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjp46\" (UniqueName: \"kubernetes.io/projected/43af9500-17cd-44ee-9625-de42cf5c4c42-kube-api-access-cjp46\") pod \"kas-connection-checker-685895c7c9-kkhpn\" (UID: \"43af9500-17cd-44ee-9625-de42cf5c4c42\") " pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" Oct 30 00:34:10.836251 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.836034 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-config\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.836251 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.836141 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wds8\" (UniqueName: \"kubernetes.io/projected/0432d327-4236-4bd4-8f39-d2dfe783e07e-kube-api-access-9wds8\") pod \"kas-connection-checker-685895c7c9-w2hfm\" (UID: \"0432d327-4236-4bd4-8f39-d2dfe783e07e\") " pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" Oct 30 00:34:10.836251 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.836169 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kf5w\" (UniqueName: \"kubernetes.io/projected/3f53e15a-0cbc-4a61-ae76-9ddc8996cabc-kube-api-access-8kf5w\") pod \"kas-connection-checker-685895c7c9-xkfl8\" (UID: \"3f53e15a-0cbc-4a61-ae76-9ddc8996cabc\") " pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" Oct 30 00:34:10.836975 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.836955 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.837062 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.837018 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.837062 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.837053 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djt7s\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.841771 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kkhpn"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.841822 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w2hfm"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.841904 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xkfl8"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.841937 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6d46m"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.841958 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.841986 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gbrdh"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842012 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77dbcdb46-8kvk8"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842034 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842141 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842168 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b6dd9b44c-l6dl6"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842195 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kcpgq"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842219 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842245 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tvh9g"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842277 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-696c77d7b7-f5pjl"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842319 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842349 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842370 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gbrwr"] Oct 30 00:34:10.842667 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.842497 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:10.893177 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893152 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 00:34:10.893312 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893232 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.893312 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893268 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 00:34:10.893312 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893285 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.893587 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893554 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.893752 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893737 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.893819 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893752 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.893819 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893732 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.893922 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893854 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.893922 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893887 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 00:34:10.893993 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.893945 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 00:34:10.894106 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.894086 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 00:34:10.894207 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.894115 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 00:34:10.894207 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.894179 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.899397 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.899377 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 00:34:10.907633 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.907601 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 00:34:10.907633 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.907618 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.907786 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.907693 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:34:10.907786 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.907705 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.907914 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.907897 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.908009 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.907997 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.908369 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.908345 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:34:10.908369 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.908347 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 00:34:10.908492 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.908459 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 00:34:10.908549 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.908517 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pkqjh\"" Oct 30 00:34:10.908607 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.908560 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.908607 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.908590 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 00:34:10.908706 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.908604 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.908706 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.908675 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 00:34:10.913778 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.913762 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:34:10.913869 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.913788 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 00:34:10.913869 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.913812 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.913869 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.913846 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.913989 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.913955 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.913989 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.913977 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 00:34:10.914059 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.913977 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.914059 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.914004 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:10.914059 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.914006 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 00:34:10.914059 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.914033 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 00:34:10.914059 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.914035 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:34:10.922760 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.922731 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-75g57\"" Oct 30 00:34:10.937494 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.937458 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbbt7\" (UniqueName: \"kubernetes.io/projected/1d35d272-2cbb-4667-beb8-252aec93d74c-kube-api-access-sbbt7\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:10.937494 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.937494 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:10.937713 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.937526 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36441a12-4446-4269-b6fd-61d82a00d242-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:10.937788 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.937765 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhd7\" (UniqueName: \"kubernetes.io/projected/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-kube-api-access-8mhd7\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:10.937838 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.937806 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d35d272-2cbb-4667-beb8-252aec93d74c-config\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:10.937894 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.937843 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:10.937941 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.937907 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4kv\" (UniqueName: \"kubernetes.io/projected/af5fef70-5659-4e51-9c0e-f23dd4e998e2-kube-api-access-5b4kv\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:10.937992 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.937949 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjp46\" (UniqueName: \"kubernetes.io/projected/43af9500-17cd-44ee-9625-de42cf5c4c42-kube-api-access-cjp46\") pod \"kas-connection-checker-685895c7c9-kkhpn\" (UID: \"43af9500-17cd-44ee-9625-de42cf5c4c42\") " pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" Oct 30 00:34:10.938041 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.937984 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/62391a80-1181-4e4b-98a2-bcc1c692f460-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:10.938041 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938019 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk566\" (UniqueName: \"kubernetes.io/projected/1c8d1f51-588d-44a5-8524-7ab17df16f1e-kube-api-access-jk566\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:10.938124 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938040 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:10.938124 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938096 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kf5w\" (UniqueName: \"kubernetes.io/projected/3f53e15a-0cbc-4a61-ae76-9ddc8996cabc-kube-api-access-8kf5w\") pod \"kas-connection-checker-685895c7c9-xkfl8\" (UID: \"3f53e15a-0cbc-4a61-ae76-9ddc8996cabc\") " pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" Oct 30 00:34:10.938215 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938130 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.938215 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938153 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-snapshots\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:10.938215 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938192 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk44v\" (UniqueName: \"kubernetes.io/projected/0203176b-a80e-4521-8bf3-833d5b973164-kube-api-access-zk44v\") pod \"volume-data-source-validator-58c9f5f46d-2snxj\" (UID: \"0203176b-a80e-4521-8bf3-833d5b973164\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" Oct 30 00:34:10.938356 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938244 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.938356 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938263 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djt7s\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.938356 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938284 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-default-certificate\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:10.938356 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938310 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-stats-auth\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:10.938527 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938357 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36441a12-4446-4269-b6fd-61d82a00d242-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:10.938527 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938408 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a5c79ec-41c6-4613-a010-af274b5f21ac-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:10.938662 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938523 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.938662 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938584 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbcq\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:10.938662 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938650 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:10.938804 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938677 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvqlt\" (UniqueName: \"kubernetes.io/projected/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-kube-api-access-wvqlt\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:10.938804 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938705 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af5fef70-5659-4e51-9c0e-f23dd4e998e2-cert\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:10.938804 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.938804 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938800 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8brp\" (UniqueName: \"kubernetes.io/projected/a2a1f298-60ee-4efe-8cec-88db4036a4e8-kube-api-access-s8brp\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.938985 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938826 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.938985 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938854 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:10.938985 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938884 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.938985 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938956 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-config-volume\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:10.939154 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.938985 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:10.939154 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939007 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8qnl\" (UniqueName: \"kubernetes.io/projected/62391a80-1181-4e4b-98a2-bcc1c692f460-kube-api-access-h8qnl\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:10.939154 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939060 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2a1f298-60ee-4efe-8cec-88db4036a4e8-serving-cert\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.939154 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939104 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.939154 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939151 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh642\" (UniqueName: \"kubernetes.io/projected/1a5c79ec-41c6-4613-a010-af274b5f21ac-kube-api-access-nh642\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:10.939351 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939194 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:10.939351 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939230 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:10.939351 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939267 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.939351 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:10.939289 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:34:10.939351 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939291 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:10.939351 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939326 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-config\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.939351 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:10.939352 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert podName:36441a12-4446-4269-b6fd-61d82a00d242 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:11.439335708 +0000 UTC m=+31.566475551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6d46m" (UID: "36441a12-4446-4269-b6fd-61d82a00d242") : secret "networking-console-plugin-cert" not found Oct 30 00:34:10.939657 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939373 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:10.939657 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939417 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:10.939657 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939451 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wds8\" (UniqueName: \"kubernetes.io/projected/0432d327-4236-4bd4-8f39-d2dfe783e07e-kube-api-access-9wds8\") pod \"kas-connection-checker-685895c7c9-w2hfm\" (UID: \"0432d327-4236-4bd4-8f39-d2dfe783e07e\") " pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" Oct 30 00:34:10.939657 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:10.939470 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:10.939657 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939552 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:10.939657 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:10.939564 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77dbcdb46-8kvk8: secret "image-registry-tls" not found Oct 30 00:34:10.939657 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939599 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:10.939657 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939619 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/104d94b9-331d-4121-9f9f-61c191d85bba-serving-cert\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:10.939657 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:10.939632 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls podName:afc047e1-508a-4f05-ae89-58db39eb5bdf nodeName:}" failed. No retries permitted until 2026-10-30 00:34:11.439615684 +0000 UTC m=+31.566755507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls") pod "image-registry-77dbcdb46-8kvk8" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf") : secret "image-registry-tls" not found Oct 30 00:34:10.940068 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939778 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sk44\" (UniqueName: \"kubernetes.io/projected/104d94b9-331d-4121-9f9f-61c191d85bba-kube-api-access-7sk44\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:10.940068 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939823 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-tmp-dir\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:10.940068 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939887 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.940068 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939920 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d35d272-2cbb-4667-beb8-252aec93d74c-serving-cert\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:10.940068 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939957 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-config\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.940068 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.939989 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:10.940267 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.940082 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:10.940267 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.940226 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:10.940335 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.940267 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a5c79ec-41c6-4613-a010-af274b5f21ac-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:10.940335 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.940295 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9t6\" (UniqueName: \"kubernetes.io/projected/54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da-kube-api-access-rj9t6\") pod \"network-check-source-6f78d59f99-s65xl\" (UID: \"54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" Oct 30 00:34:10.940406 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.940361 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-tmp\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:10.940514 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.940491 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.942506 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.942479 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 00:34:10.942728 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.942709 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2a1f298-60ee-4efe-8cec-88db4036a4e8-serving-cert\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.942788 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.942765 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.942849 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.942786 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.950484 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.950463 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 00:34:10.950608 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.950502 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.951152 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.951136 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 00:34:10.959850 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.959799 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.963453 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.963386 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjp46\" (UniqueName: \"kubernetes.io/projected/43af9500-17cd-44ee-9625-de42cf5c4c42-kube-api-access-cjp46\") pod \"kas-connection-checker-685895c7c9-kkhpn\" (UID: \"43af9500-17cd-44ee-9625-de42cf5c4c42\") " pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" Oct 30 00:34:10.963607 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.963556 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kf5w\" (UniqueName: \"kubernetes.io/projected/3f53e15a-0cbc-4a61-ae76-9ddc8996cabc-kube-api-access-8kf5w\") pod \"kas-connection-checker-685895c7c9-xkfl8\" (UID: \"3f53e15a-0cbc-4a61-ae76-9ddc8996cabc\") " pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" Oct 30 00:34:10.963659 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.963636 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djt7s\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.964104 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.964076 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:10.964505 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.964490 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8brp\" (UniqueName: \"kubernetes.io/projected/a2a1f298-60ee-4efe-8cec-88db4036a4e8-kube-api-access-s8brp\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:10.965013 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:10.964997 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wds8\" (UniqueName: \"kubernetes.io/projected/0432d327-4236-4bd4-8f39-d2dfe783e07e-kube-api-access-9wds8\") pod \"kas-connection-checker-685895c7c9-w2hfm\" (UID: \"0432d327-4236-4bd4-8f39-d2dfe783e07e\") " pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" Oct 30 00:34:11.040866 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.040828 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.040866 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.040880 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:11.041066 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.040975 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.041066 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.040994 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a5c79ec-41c6-4613-a010-af274b5f21ac-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:11.041066 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041022 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9t6\" (UniqueName: \"kubernetes.io/projected/54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da-kube-api-access-rj9t6\") pod \"network-check-source-6f78d59f99-s65xl\" (UID: \"54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" Oct 30 00:34:11.041066 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041048 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-tmp\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.041348 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.041092 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:34:11.041348 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.041184 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls podName:1c8d1f51-588d-44a5-8524-7ab17df16f1e nodeName:}" failed. No retries permitted until 2026-10-30 00:34:11.541163405 +0000 UTC m=+31.668303242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4xbzc" (UID: "1c8d1f51-588d-44a5-8524-7ab17df16f1e") : secret "samples-operator-tls" not found Oct 30 00:34:11.041348 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041211 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbt7\" (UniqueName: \"kubernetes.io/projected/1d35d272-2cbb-4667-beb8-252aec93d74c-kube-api-access-sbbt7\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:11.041348 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041246 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.041348 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041276 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhd7\" (UniqueName: \"kubernetes.io/projected/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-kube-api-access-8mhd7\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.041348 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041301 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d35d272-2cbb-4667-beb8-252aec93d74c-config\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:11.041348 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041328 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041354 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4kv\" (UniqueName: \"kubernetes.io/projected/af5fef70-5659-4e51-9c0e-f23dd4e998e2-kube-api-access-5b4kv\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041383 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/62391a80-1181-4e4b-98a2-bcc1c692f460-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041412 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk566\" (UniqueName: \"kubernetes.io/projected/1c8d1f51-588d-44a5-8524-7ab17df16f1e-kube-api-access-jk566\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041434 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041462 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-snapshots\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.041491 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:11.54147167 +0000 UTC m=+31.668611509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : configmap references non-existent config key: service-ca.crt Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041538 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk44v\" (UniqueName: \"kubernetes.io/projected/0203176b-a80e-4521-8bf3-833d5b973164-kube-api-access-zk44v\") pod \"volume-data-source-validator-58c9f5f46d-2snxj\" (UID: \"0203176b-a80e-4521-8bf3-833d5b973164\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041599 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-default-certificate\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041627 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-stats-auth\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041653 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a5c79ec-41c6-4613-a010-af274b5f21ac-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041706 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbcq\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041731 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041739 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a5c79ec-41c6-4613-a010-af274b5f21ac-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041760 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvqlt\" (UniqueName: \"kubernetes.io/projected/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-kube-api-access-wvqlt\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:11.042075 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041799 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af5fef70-5659-4e51-9c0e-f23dd4e998e2-cert\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041837 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041864 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-config-volume\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041892 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041925 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8qnl\" (UniqueName: \"kubernetes.io/projected/62391a80-1181-4e4b-98a2-bcc1c692f460-kube-api-access-h8qnl\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041959 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh642\" (UniqueName: \"kubernetes.io/projected/1a5c79ec-41c6-4613-a010-af274b5f21ac-kube-api-access-nh642\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042010 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.042024 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042052 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.042069 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:11.542055222 +0000 UTC m=+31.669195076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042103 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042133 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042168 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042197 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.042204 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042222 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/104d94b9-331d-4121-9f9f-61c191d85bba-serving-cert\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.043980 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042242 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/62391a80-1181-4e4b-98a2-bcc1c692f460-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.042254 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls podName:62391a80-1181-4e4b-98a2-bcc1c692f460 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:11.542239788 +0000 UTC m=+31.669379610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7b7wb" (UID: "62391a80-1181-4e4b-98a2-bcc1c692f460") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042278 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sk44\" (UniqueName: \"kubernetes.io/projected/104d94b9-331d-4121-9f9f-61c191d85bba-kube-api-access-7sk44\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042312 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-tmp-dir\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042348 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d35d272-2cbb-4667-beb8-252aec93d74c-serving-cert\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042519 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042565 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-snapshots\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.042794 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d35d272-2cbb-4667-beb8-252aec93d74c-config\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.041497 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-tmp\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.043233 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.043317 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.043362 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:11.543346919 +0000 UTC m=+31.670486758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.043742 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.043919 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-config-volume\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.044053 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.044144 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:11.044765 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.044157 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b6dd9b44c-l6dl6: secret "image-registry-tls" not found Oct 30 00:34:11.045471 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.044169 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.045471 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.044197 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls podName:89b792bc-b5bc-4153-86dc-65f1b7187a5a nodeName:}" failed. No retries permitted until 2026-10-30 00:34:11.544184164 +0000 UTC m=+31.671324001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls") pod "image-registry-6b6dd9b44c-l6dl6" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a") : secret "image-registry-tls" not found Oct 30 00:34:11.045471 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.044432 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-tmp-dir\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:11.045471 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.045001 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.045471 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.045302 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d35d272-2cbb-4667-beb8-252aec93d74c-serving-cert\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:11.045471 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.045464 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.045471 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.045477 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/104d94b9-331d-4121-9f9f-61c191d85bba-serving-cert\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.045769 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.045541 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af5fef70-5659-4e51-9c0e-f23dd4e998e2-cert\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:11.047014 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.046988 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-stats-auth\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.047140 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.047118 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-default-certificate\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.047507 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.047481 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a5c79ec-41c6-4613-a010-af274b5f21ac-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:11.048266 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.048249 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.069267 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.069234 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbt7\" (UniqueName: \"kubernetes.io/projected/1d35d272-2cbb-4667-beb8-252aec93d74c-kube-api-access-sbbt7\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:11.069446 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.069298 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9t6\" (UniqueName: \"kubernetes.io/projected/54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da-kube-api-access-rj9t6\") pod \"network-check-source-6f78d59f99-s65xl\" (UID: \"54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" Oct 30 00:34:11.069728 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.069710 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.071348 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.071328 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4kv\" (UniqueName: \"kubernetes.io/projected/af5fef70-5659-4e51-9c0e-f23dd4e998e2-kube-api-access-5b4kv\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:11.081844 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.081823 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" Oct 30 00:34:11.088756 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.088733 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" Oct 30 00:34:11.090935 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.090912 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhd7\" (UniqueName: \"kubernetes.io/projected/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-kube-api-access-8mhd7\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.094606 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.094139 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbcq\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.094606 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.094264 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvqlt\" (UniqueName: \"kubernetes.io/projected/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-kube-api-access-wvqlt\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:11.094606 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.094294 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk44v\" (UniqueName: \"kubernetes.io/projected/0203176b-a80e-4521-8bf3-833d5b973164-kube-api-access-zk44v\") pod \"volume-data-source-validator-58c9f5f46d-2snxj\" (UID: \"0203176b-a80e-4521-8bf3-833d5b973164\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" Oct 30 00:34:11.094606 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.094431 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8qnl\" (UniqueName: \"kubernetes.io/projected/62391a80-1181-4e4b-98a2-bcc1c692f460-kube-api-access-h8qnl\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:11.094756 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.094672 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh642\" (UniqueName: \"kubernetes.io/projected/1a5c79ec-41c6-4613-a010-af274b5f21ac-kube-api-access-nh642\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:11.094756 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.094707 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sk44\" (UniqueName: \"kubernetes.io/projected/104d94b9-331d-4121-9f9f-61c191d85bba-kube-api-access-7sk44\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.095592 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.094857 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk566\" (UniqueName: \"kubernetes.io/projected/1c8d1f51-588d-44a5-8524-7ab17df16f1e-kube-api-access-jk566\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:11.097931 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.097908 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" Oct 30 00:34:11.132599 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.132551 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:11.152832 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.152804 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:11.164781 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.164759 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" Oct 30 00:34:11.170453 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.170431 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:11.188735 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.183064 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:11.198920 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.198467 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" Oct 30 00:34:11.213678 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.212921 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:11.447163 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.447134 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:11.447163 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.447166 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:11.447662 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.447374 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:11.447662 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.447394 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:34:11.447662 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.447471 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert podName:36441a12-4446-4269-b6fd-61d82a00d242 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.447436331 +0000 UTC m=+32.574576155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6d46m" (UID: "36441a12-4446-4269-b6fd-61d82a00d242") : secret "networking-console-plugin-cert" not found Oct 30 00:34:11.447662 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.447398 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77dbcdb46-8kvk8: secret "image-registry-tls" not found Oct 30 00:34:11.447662 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.447510 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls podName:afc047e1-508a-4f05-ae89-58db39eb5bdf nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.447501491 +0000 UTC m=+32.574641312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls") pod "image-registry-77dbcdb46-8kvk8" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf") : secret "image-registry-tls" not found Oct 30 00:34:11.451535 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.451516 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xkfl8"] Oct 30 00:34:11.455232 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:11.455191 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f53e15a_0cbc_4a61_ae76_9ddc8996cabc.slice/crio-a6f6ba7f4d25e13d4b30bd1e308d93b78c0a53a27f24f5683c5127952715ba3c WatchSource:0}: Error finding container a6f6ba7f4d25e13d4b30bd1e308d93b78c0a53a27f24f5683c5127952715ba3c: Status 404 returned error can't find the container with id a6f6ba7f4d25e13d4b30bd1e308d93b78c0a53a27f24f5683c5127952715ba3c Oct 30 00:34:11.456077 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.456046 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w2hfm"] Oct 30 00:34:11.457859 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.457773 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kkhpn"] Oct 30 00:34:11.488621 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.483193 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gbrdh"] Oct 30 00:34:11.488621 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:11.487632 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a1f298_60ee_4efe_8cec_88db4036a4e8.slice/crio-f2cf90b0b22e22363739b21241acb788719ebe7973b5ca57e50dacb54583073c WatchSource:0}: Error finding container f2cf90b0b22e22363739b21241acb788719ebe7973b5ca57e50dacb54583073c: Status 404 returned error can't find the container with id f2cf90b0b22e22363739b21241acb788719ebe7973b5ca57e50dacb54583073c Oct 30 00:34:11.510207 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.510180 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tvh9g"] Oct 30 00:34:11.521073 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:11.518925 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104d94b9_331d_4121_9f9f_61c191d85bba.slice/crio-0de166975d128aded3385a28ba7be67bbbb5049da2739a6c782ec5773c353286 WatchSource:0}: Error finding container 0de166975d128aded3385a28ba7be67bbbb5049da2739a6c782ec5773c353286: Status 404 returned error can't find the container with id 0de166975d128aded3385a28ba7be67bbbb5049da2739a6c782ec5773c353286 Oct 30 00:34:11.547862 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.547815 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:11.548049 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.547893 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:11.548049 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.547947 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.548049 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.547986 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:11.548221 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.548053 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:11.548221 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.548093 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:11.548316 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548284 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:11.548370 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548345 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.5483313 +0000 UTC m=+32.675471121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:11.548431 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548412 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:11.548463 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548424 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b6dd9b44c-l6dl6: secret "image-registry-tls" not found Oct 30 00:34:11.548491 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548470 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls podName:89b792bc-b5bc-4153-86dc-65f1b7187a5a nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.548450532 +0000 UTC m=+32.675590353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls") pod "image-registry-6b6dd9b44c-l6dl6" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a") : secret "image-registry-tls" not found Oct 30 00:34:11.548542 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548524 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:34:11.548676 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548560 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls podName:1c8d1f51-588d-44a5-8524-7ab17df16f1e nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.548548997 +0000 UTC m=+32.675688816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4xbzc" (UID: "1c8d1f51-588d-44a5-8524-7ab17df16f1e") : secret "samples-operator-tls" not found Oct 30 00:34:11.548676 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548629 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:11.548676 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548659 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.548649626 +0000 UTC m=+32.675789462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:11.548676 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548654 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:11.548922 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548703 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.548682049 +0000 UTC m=+32.675821886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : configmap references non-existent config key: service-ca.crt Oct 30 00:34:11.548922 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:11.548731 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls podName:62391a80-1181-4e4b-98a2-bcc1c692f460 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.548720408 +0000 UTC m=+32.675860228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7b7wb" (UID: "62391a80-1181-4e4b-98a2-bcc1c692f460") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:11.562092 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.562060 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj"] Oct 30 00:34:11.567741 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:11.567712 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0203176b_a80e_4521_8bf3_833d5b973164.slice/crio-f6f2c6f0d95c8da27a44e1dd4838c6b176ba1bd571c12176bd65232f97bef791 WatchSource:0}: Error finding container f6f2c6f0d95c8da27a44e1dd4838c6b176ba1bd571c12176bd65232f97bef791: Status 404 returned error can't find the container with id f6f2c6f0d95c8da27a44e1dd4838c6b176ba1bd571c12176bd65232f97bef791 Oct 30 00:34:11.591628 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.591603 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79"] Oct 30 00:34:11.591810 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.591779 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" event={"ID":"a2a1f298-60ee-4efe-8cec-88db4036a4e8","Type":"ContainerStarted","Data":"f2cf90b0b22e22363739b21241acb788719ebe7973b5ca57e50dacb54583073c"} Oct 30 00:34:11.593018 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.592992 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" event={"ID":"0203176b-a80e-4521-8bf3-833d5b973164","Type":"ContainerStarted","Data":"f6f2c6f0d95c8da27a44e1dd4838c6b176ba1bd571c12176bd65232f97bef791"} Oct 30 00:34:11.593823 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:11.593801 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5c79ec_41c6_4613_a010_af274b5f21ac.slice/crio-362559db3e6cf19edbb4becaf195b300b35557ab936caddeeaa22f73ec146a34 WatchSource:0}: Error finding container 362559db3e6cf19edbb4becaf195b300b35557ab936caddeeaa22f73ec146a34: Status 404 returned error can't find the container with id 362559db3e6cf19edbb4becaf195b300b35557ab936caddeeaa22f73ec146a34 Oct 30 00:34:11.594371 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.594206 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tvh9g" event={"ID":"104d94b9-331d-4121-9f9f-61c191d85bba","Type":"ContainerStarted","Data":"0de166975d128aded3385a28ba7be67bbbb5049da2739a6c782ec5773c353286"} Oct 30 00:34:11.595402 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.595380 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" event={"ID":"0432d327-4236-4bd4-8f39-d2dfe783e07e","Type":"ContainerStarted","Data":"bf692c2f892512f43b57d3865d6b479a21e1ae5b94755c04ef8c8f4c7d9390c0"} Oct 30 00:34:11.596345 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.596325 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" event={"ID":"3f53e15a-0cbc-4a61-ae76-9ddc8996cabc","Type":"ContainerStarted","Data":"a6f6ba7f4d25e13d4b30bd1e308d93b78c0a53a27f24f5683c5127952715ba3c"} Oct 30 00:34:11.597314 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.597297 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" event={"ID":"43af9500-17cd-44ee-9625-de42cf5c4c42","Type":"ContainerStarted","Data":"0a408f6c504d3b1bf82b07655d73f28020e68df38b6e6b2cea42f7959e75a4c7"} Oct 30 00:34:11.616492 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.616401 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl"] Oct 30 00:34:11.618889 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:11.618860 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f5b3b0_d74a_4d7e_9cf1_1fbbf65ee7da.slice/crio-5f839982ed877b3e784d19141d8d84fe90aafc67b58cd9f2fc558b9bf815186a WatchSource:0}: Error finding container 5f839982ed877b3e784d19141d8d84fe90aafc67b58cd9f2fc558b9bf815186a: Status 404 returned error can't find the container with id 5f839982ed877b3e784d19141d8d84fe90aafc67b58cd9f2fc558b9bf815186a Oct 30 00:34:11.619205 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.619185 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj"] Oct 30 00:34:11.623477 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:11.622527 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d35d272_2cbb_4667_beb8_252aec93d74c.slice/crio-d9c61d4151706da569456cd8d3810924fa0b567f3dfcee493090a4f690c8307f WatchSource:0}: Error finding container d9c61d4151706da569456cd8d3810924fa0b567f3dfcee493090a4f690c8307f: Status 404 returned error can't find the container with id d9c61d4151706da569456cd8d3810924fa0b567f3dfcee493090a4f690c8307f Oct 30 00:34:11.662231 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:11.662203 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kcpgq"] Oct 30 00:34:11.664658 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:11.664611 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5fef70_5659_4e51_9c0e_f23dd4e998e2.slice/crio-c346430025e6eb00276a65082ab365066ebfcd79c5d47ec98666c235683eaf29 WatchSource:0}: Error finding container c346430025e6eb00276a65082ab365066ebfcd79c5d47ec98666c235683eaf29: Status 404 returned error can't find the container with id c346430025e6eb00276a65082ab365066ebfcd79c5d47ec98666c235683eaf29 Oct 30 00:34:12.377686 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.377650 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:12.378143 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.378123 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:12.378505 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.378487 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:12.397377 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.397344 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-smldl\"" Oct 30 00:34:12.397547 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.397467 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78zxw\"" Oct 30 00:34:12.397786 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.397766 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:34:12.397876 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.397840 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q972j\"" Oct 30 00:34:12.398155 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.398124 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:34:12.458323 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.457457 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:12.458323 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.457505 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:12.458323 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.457730 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:12.458323 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.457746 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77dbcdb46-8kvk8: secret "image-registry-tls" not found Oct 30 00:34:12.458323 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.457807 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls podName:afc047e1-508a-4f05-ae89-58db39eb5bdf nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.457787176 +0000 UTC m=+34.584927002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls") pod "image-registry-77dbcdb46-8kvk8" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf") : secret "image-registry-tls" not found Oct 30 00:34:12.458323 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.458228 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:34:12.458323 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.458279 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert podName:36441a12-4446-4269-b6fd-61d82a00d242 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.45826481 +0000 UTC m=+34.585404636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6d46m" (UID: "36441a12-4446-4269-b6fd-61d82a00d242") : secret "networking-console-plugin-cert" not found Oct 30 00:34:12.558279 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.558236 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:12.558481 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.558310 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:12.558481 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.558358 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:12.558481 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.558396 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:12.558481 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.558461 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:12.558838 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.558491 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:12.558838 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.558684 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:12.558838 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.558750 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.558730504 +0000 UTC m=+34.685870341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559509 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559529 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b6dd9b44c-l6dl6: secret "image-registry-tls" not found Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559590 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls podName:89b792bc-b5bc-4153-86dc-65f1b7187a5a nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.559558665 +0000 UTC m=+34.686698492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls") pod "image-registry-6b6dd9b44c-l6dl6" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a") : secret "image-registry-tls" not found Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559656 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559692 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls podName:1c8d1f51-588d-44a5-8524-7ab17df16f1e nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.559678863 +0000 UTC m=+34.686818687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4xbzc" (UID: "1c8d1f51-588d-44a5-8524-7ab17df16f1e") : secret "samples-operator-tls" not found Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559758 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.559745969 +0000 UTC m=+34.686885795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : configmap references non-existent config key: service-ca.crt Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559811 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559844 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls podName:62391a80-1181-4e4b-98a2-bcc1c692f460 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.559833904 +0000 UTC m=+34.686973725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7b7wb" (UID: "62391a80-1181-4e4b-98a2-bcc1c692f460") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559896 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:12.560194 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:12.559933 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:14.559923661 +0000 UTC m=+34.687063483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:12.605596 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.605528 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" event={"ID":"0432d327-4236-4bd4-8f39-d2dfe783e07e","Type":"ContainerStarted","Data":"bbafa6d655bcf12b460ec65ca5cafb1ede66eab394dd6986c13ace7b98d74025"} Oct 30 00:34:12.611108 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.611065 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" event={"ID":"3f53e15a-0cbc-4a61-ae76-9ddc8996cabc","Type":"ContainerStarted","Data":"a87831bb7e8b583eb0c3ab41874f2197ab188c8c943f41635ae9ffaa781a4edc"} Oct 30 00:34:12.613584 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.613499 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" event={"ID":"43af9500-17cd-44ee-9625-de42cf5c4c42","Type":"ContainerStarted","Data":"5941174b74f1ab1708c4c922af1572853e8e0ed706bc3037f07ede539db48f03"} Oct 30 00:34:12.615695 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.615601 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kcpgq" event={"ID":"af5fef70-5659-4e51-9c0e-f23dd4e998e2","Type":"ContainerStarted","Data":"c346430025e6eb00276a65082ab365066ebfcd79c5d47ec98666c235683eaf29"} Oct 30 00:34:12.619520 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.619464 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" event={"ID":"54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da","Type":"ContainerStarted","Data":"5f839982ed877b3e784d19141d8d84fe90aafc67b58cd9f2fc558b9bf815186a"} Oct 30 00:34:12.626659 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.626588 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" event={"ID":"1d35d272-2cbb-4667-beb8-252aec93d74c","Type":"ContainerStarted","Data":"d9c61d4151706da569456cd8d3810924fa0b567f3dfcee493090a4f690c8307f"} Oct 30 00:34:12.629547 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.629483 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" event={"ID":"1a5c79ec-41c6-4613-a010-af274b5f21ac","Type":"ContainerStarted","Data":"362559db3e6cf19edbb4becaf195b300b35557ab936caddeeaa22f73ec146a34"} Oct 30 00:34:12.651004 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.650379 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" podStartSLOduration=401.650358772 podStartE2EDuration="6m41.650358772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:12.648989234 +0000 UTC m=+32.776129077" watchObservedRunningTime="2026-10-30 00:34:12.650358772 +0000 UTC m=+32.777498616" Oct 30 00:34:12.741981 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:12.740556 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" podStartSLOduration=401.740536477 podStartE2EDuration="6m41.740536477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:12.696744625 +0000 UTC m=+32.823884472" watchObservedRunningTime="2026-10-30 00:34:12.740536477 +0000 UTC m=+32.867676322" Oct 30 00:34:14.180957 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.180324 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:14.180957 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.180455 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:14.180957 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.180623 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:14.180957 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.180691 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:46.180671652 +0000 UTC m=+66.307811473 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : secret "metrics-daemon-secret" not found Oct 30 00:34:14.191005 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.186294 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:14.260089 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.260053 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:14.281403 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.281370 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:14.285147 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.285080 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:14.486228 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.486025 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:14.486228 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.486066 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:14.486228 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.486183 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:34:14.486505 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.486257 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert podName:36441a12-4446-4269-b6fd-61d82a00d242 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.486238691 +0000 UTC m=+38.613378525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6d46m" (UID: "36441a12-4446-4269-b6fd-61d82a00d242") : secret "networking-console-plugin-cert" not found Oct 30 00:34:14.486691 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.486186 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:14.486691 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.486635 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77dbcdb46-8kvk8: secret "image-registry-tls" not found Oct 30 00:34:14.486838 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.486697 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls podName:afc047e1-508a-4f05-ae89-58db39eb5bdf nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.486678971 +0000 UTC m=+38.613818807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls") pod "image-registry-77dbcdb46-8kvk8" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf") : secret "image-registry-tls" not found Oct 30 00:34:14.537449 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.537414 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.586684 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.586747 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.586813 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.586845 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.586917 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.586958 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587101 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587166 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls podName:1c8d1f51-588d-44a5-8524-7ab17df16f1e nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.587149164 +0000 UTC m=+38.714289004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4xbzc" (UID: "1c8d1f51-588d-44a5-8524-7ab17df16f1e") : secret "samples-operator-tls" not found Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587594 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.58756101 +0000 UTC m=+38.714700848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : configmap references non-existent config key: service-ca.crt Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587667 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587699 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls podName:62391a80-1181-4e4b-98a2-bcc1c692f460 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.587691259 +0000 UTC m=+38.714831092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7b7wb" (UID: "62391a80-1181-4e4b-98a2-bcc1c692f460") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587745 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587765 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.587759433 +0000 UTC m=+38.714899253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:14.587786 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587783 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:14.588652 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587848 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.587830441 +0000 UTC m=+38.714970276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:14.588652 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587883 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:14.588652 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587898 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b6dd9b44c-l6dl6: secret "image-registry-tls" not found Oct 30 00:34:14.588652 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:14.587938 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls podName:89b792bc-b5bc-4153-86dc-65f1b7187a5a nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.587927912 +0000 UTC m=+38.715067745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls") pod "image-registry-6b6dd9b44c-l6dl6" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a") : secret "image-registry-tls" not found Oct 30 00:34:14.620213 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.620140 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" podStartSLOduration=403.620120617 podStartE2EDuration="6m43.620120617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:12.741955883 +0000 UTC m=+32.869095726" watchObservedRunningTime="2026-10-30 00:34:14.620120617 +0000 UTC m=+34.747260462" Oct 30 00:34:14.621093 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.621065 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rkgf"] Oct 30 00:34:14.688127 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.688027 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rkgf" event={"ID":"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec","Type":"ContainerStarted","Data":"bbc6d4e5e1d7ad00156ba4c956e0e82dd18a15363c35cd46fe1a2f47340c24d4"} Oct 30 00:34:14.692462 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.691816 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" event={"ID":"0203176b-a80e-4521-8bf3-833d5b973164","Type":"ContainerStarted","Data":"764b03c8b2957c48c04593f03571336475b8eb7ccec1ac74feaa2a5074a5c771"} Oct 30 00:34:14.935091 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.934972 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" podStartSLOduration=371.27224633 podStartE2EDuration="6m13.934950086s" totalImagesPullingTime="2.662703756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:14.737924142 +0000 UTC m=+34.865063983" watchObservedRunningTime="2026-10-30 00:34:14.934950086 +0000 UTC m=+35.062089933" Oct 30 00:34:14.936020 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:14.935995 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wzp7p"] Oct 30 00:34:14.941000 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:14.940957 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d4ec0d_58d5_4556_a9dc_ddb087fbd1e6.slice/crio-02822fb876f7b1af92b54160650e617340f64a7fdb6739666890a6177293ed0c WatchSource:0}: Error finding container 02822fb876f7b1af92b54160650e617340f64a7fdb6739666890a6177293ed0c: Status 404 returned error can't find the container with id 02822fb876f7b1af92b54160650e617340f64a7fdb6739666890a6177293ed0c Oct 30 00:34:15.698803 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:15.698768 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wzp7p" event={"ID":"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6","Type":"ContainerStarted","Data":"02822fb876f7b1af92b54160650e617340f64a7fdb6739666890a6177293ed0c"} Oct 30 00:34:16.709477 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:16.709410 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/0.log" Oct 30 00:34:16.709477 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:16.709453 2584 generic.go:356] "Generic (PLEG): container finished" podID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" containerID="d7acc1732d2fef50fe69da7b5c5f917b1a0408121e7123295cca93ccc7a66a2d" exitCode=255 Oct 30 00:34:16.709871 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:16.709534 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" event={"ID":"a2a1f298-60ee-4efe-8cec-88db4036a4e8","Type":"ContainerDied","Data":"d7acc1732d2fef50fe69da7b5c5f917b1a0408121e7123295cca93ccc7a66a2d"} Oct 30 00:34:16.710287 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:16.710264 2584 scope.go:122] "RemoveContainer" containerID="d7acc1732d2fef50fe69da7b5c5f917b1a0408121e7123295cca93ccc7a66a2d" Oct 30 00:34:16.713215 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:16.712716 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" event={"ID":"1a5c79ec-41c6-4613-a010-af274b5f21ac","Type":"ContainerStarted","Data":"8e518e24bee9cb7c0a1f60a131e0a68122fa3bed923d2b72d186a7ed67481463"} Oct 30 00:34:16.715360 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:16.715332 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tvh9g" event={"ID":"104d94b9-331d-4121-9f9f-61c191d85bba","Type":"ContainerStarted","Data":"592d79ffa2135db3907ea3ff50f370305530ec117cdb1719589ba7c5bf937e35"} Oct 30 00:34:16.805888 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:16.805523 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tvh9g" podStartSLOduration=374.964682396 podStartE2EDuration="6m19.805501174s" totalImagesPullingTime="4.840818778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:16.803479969 +0000 UTC m=+36.930619815" watchObservedRunningTime="2026-10-30 00:34:16.805501174 +0000 UTC m=+36.932641018" Oct 30 00:34:17.398746 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.398638 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" podStartSLOduration=371.64213939 podStartE2EDuration="6m16.398620107s" totalImagesPullingTime="4.756480717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:16.847643843 +0000 UTC m=+36.974783685" watchObservedRunningTime="2026-10-30 00:34:17.398620107 +0000 UTC m=+37.525759982" Oct 30 00:34:17.399314 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.399288 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj"] Oct 30 00:34:17.415969 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.415857 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" Oct 30 00:34:17.419914 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.419844 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj"] Oct 30 00:34:17.505478 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.504065 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9pkrm"] Oct 30 00:34:17.519382 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.519348 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:17.519591 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.519559 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 00:34:17.519683 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.519612 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-th8hp\"" Oct 30 00:34:17.523714 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.523685 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.528902 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.528876 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljjz\" (UniqueName: \"kubernetes.io/projected/d585123a-11a7-4f0f-9ae7-823a4becc3f1-kube-api-access-jljjz\") pod \"migrator-66df5d496b-z7jzj\" (UID: \"d585123a-11a7-4f0f-9ae7-823a4becc3f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" Oct 30 00:34:17.530501 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.530477 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9pkrm"] Oct 30 00:34:17.619412 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.619384 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:34:17.619652 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.619469 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:34:17.619652 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.619591 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c889x\"" Oct 30 00:34:17.630238 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.630199 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.630418 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.630392 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-crio-socket\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.630490 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.630441 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljjz\" (UniqueName: \"kubernetes.io/projected/d585123a-11a7-4f0f-9ae7-823a4becc3f1-kube-api-access-jljjz\") pod \"migrator-66df5d496b-z7jzj\" (UID: \"d585123a-11a7-4f0f-9ae7-823a4becc3f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" Oct 30 00:34:17.630548 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.630495 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-data-volume\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.630616 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.630566 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.630662 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.630649 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtp2\" (UniqueName: \"kubernetes.io/projected/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-api-access-hdtp2\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.656179 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.656109 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljjz\" (UniqueName: \"kubernetes.io/projected/d585123a-11a7-4f0f-9ae7-823a4becc3f1-kube-api-access-jljjz\") pod \"migrator-66df5d496b-z7jzj\" (UID: \"d585123a-11a7-4f0f-9ae7-823a4becc3f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" Oct 30 00:34:17.724151 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.724115 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/1.log" Oct 30 00:34:17.724722 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.724702 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/0.log" Oct 30 00:34:17.724788 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.724733 2584 generic.go:356] "Generic (PLEG): container finished" podID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" containerID="d0ce7c2a7fb68c749ef8342679ba6e2cf480d16cba39678ca62a829b5ebbf483" exitCode=255 Oct 30 00:34:17.724884 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.724833 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" event={"ID":"a2a1f298-60ee-4efe-8cec-88db4036a4e8","Type":"ContainerDied","Data":"d0ce7c2a7fb68c749ef8342679ba6e2cf480d16cba39678ca62a829b5ebbf483"} Oct 30 00:34:17.724884 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.724880 2584 scope.go:122] "RemoveContainer" containerID="d7acc1732d2fef50fe69da7b5c5f917b1a0408121e7123295cca93ccc7a66a2d" Oct 30 00:34:17.725666 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.725254 2584 scope.go:122] "RemoveContainer" containerID="d0ce7c2a7fb68c749ef8342679ba6e2cf480d16cba39678ca62a829b5ebbf483" Oct 30 00:34:17.725666 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:17.725459 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gbrdh_openshift-console-operator(a2a1f298-60ee-4efe-8cec-88db4036a4e8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" podUID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" Oct 30 00:34:17.728292 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.728266 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" event={"ID":"1d35d272-2cbb-4667-beb8-252aec93d74c","Type":"ContainerStarted","Data":"efe0f54ce11ba31f3ed854c95f082fa1b61c9d0ecc7969084fa4e9876da6cbe7"} Oct 30 00:34:17.731546 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.731520 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-crio-socket\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.731672 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.731613 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-data-volume\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.731739 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.731674 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.732303 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.731882 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtp2\" (UniqueName: \"kubernetes.io/projected/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-api-access-hdtp2\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.732303 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.731955 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.732303 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:17.732044 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:34:17.732303 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:17.732098 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.232079361 +0000 UTC m=+38.359219205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : configmap "kube-rbac-proxy" not found Oct 30 00:34:17.732303 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.732159 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-crio-socket\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.732303 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:17.732240 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:17.732303 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:17.732279 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:18.232266632 +0000 UTC m=+38.359406466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:17.732950 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.732600 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-data-volume\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:17.752068 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.752036 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" Oct 30 00:34:17.757605 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:17.757555 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtp2\" (UniqueName: \"kubernetes.io/projected/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-api-access-hdtp2\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:18.104417 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.104309 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" podStartSLOduration=370.968945651 podStartE2EDuration="6m16.104286573s" totalImagesPullingTime="5.135340922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:17.8155884 +0000 UTC m=+37.942728236" watchObservedRunningTime="2026-10-30 00:34:18.104286573 +0000 UTC m=+38.231426415" Oct 30 00:34:18.104940 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.104911 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj"] Oct 30 00:34:18.108098 ip-10-0-139-66 kubenswrapper[2584]: W1030 00:34:18.108069 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd585123a_11a7_4f0f_9ae7_823a4becc3f1.slice/crio-2feb38d5e4474a93a8d2d0250d088a2299e16f1ec3a5ff3d9a07a6cffc353c9a WatchSource:0}: Error finding container 2feb38d5e4474a93a8d2d0250d088a2299e16f1ec3a5ff3d9a07a6cffc353c9a: Status 404 returned error can't find the container with id 2feb38d5e4474a93a8d2d0250d088a2299e16f1ec3a5ff3d9a07a6cffc353c9a Oct 30 00:34:18.237626 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.237183 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:18.237626 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.237304 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:18.237626 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.237396 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:34:18.237626 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.237458 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:19.237440859 +0000 UTC m=+39.364580694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : configmap "kube-rbac-proxy" not found Oct 30 00:34:18.237626 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.237542 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:18.237626 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.237596 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:19.237563892 +0000 UTC m=+39.364703725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:18.540289 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.539779 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:18.540289 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.539839 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:18.540289 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.539926 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:34:18.540289 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.539991 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:18.540289 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.540005 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77dbcdb46-8kvk8: secret "image-registry-tls" not found Oct 30 00:34:18.540289 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.539993 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert podName:36441a12-4446-4269-b6fd-61d82a00d242 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.539975184 +0000 UTC m=+46.667115025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6d46m" (UID: "36441a12-4446-4269-b6fd-61d82a00d242") : secret "networking-console-plugin-cert" not found Oct 30 00:34:18.540289 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.540056 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls podName:afc047e1-508a-4f05-ae89-58db39eb5bdf nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.54004122 +0000 UTC m=+46.667181047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls") pod "image-registry-77dbcdb46-8kvk8" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf") : secret "image-registry-tls" not found Oct 30 00:34:18.640375 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.640334 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:18.640559 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.640406 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:18.640559 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.640452 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:18.640559 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.640483 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:18.640559 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640503 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:18.640559 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640529 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b6dd9b44c-l6dl6: secret "image-registry-tls" not found Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640606 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.64058585 +0000 UTC m=+46.767725693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : configmap references non-existent config key: service-ca.crt Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640605 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640628 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls podName:89b792bc-b5bc-4153-86dc-65f1b7187a5a nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.640619387 +0000 UTC m=+46.767759219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls") pod "image-registry-6b6dd9b44c-l6dl6" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a") : secret "image-registry-tls" not found Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.640649 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640672 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls podName:62391a80-1181-4e4b-98a2-bcc1c692f460 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.640658304 +0000 UTC m=+46.767798137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7b7wb" (UID: "62391a80-1181-4e4b-98a2-bcc1c692f460") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.640703 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640716 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640748 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.640739227 +0000 UTC m=+46.767879050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640605 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640781 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls podName:1c8d1f51-588d-44a5-8524-7ab17df16f1e nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.64077144 +0000 UTC m=+46.767911308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4xbzc" (UID: "1c8d1f51-588d-44a5-8524-7ab17df16f1e") : secret "samples-operator-tls" not found Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640786 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:18.640809 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.640811 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.640803545 +0000 UTC m=+46.767943366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:18.744220 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.744134 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" event={"ID":"54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da","Type":"ContainerStarted","Data":"910fc199c67ed6b7685c6c9ae3501db9c0eae0e22fa1857a8c9574ea66f74454"} Oct 30 00:34:18.746736 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.746695 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/1.log" Oct 30 00:34:18.747092 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.747070 2584 scope.go:122] "RemoveContainer" containerID="d0ce7c2a7fb68c749ef8342679ba6e2cf480d16cba39678ca62a829b5ebbf483" Oct 30 00:34:18.747280 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:18.747260 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gbrdh_openshift-console-operator(a2a1f298-60ee-4efe-8cec-88db4036a4e8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" podUID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" Oct 30 00:34:18.749190 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.749132 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wzp7p" event={"ID":"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6","Type":"ContainerStarted","Data":"8634fc72ec47b02a761f5a81d8dbf7b7e4894459a07686785441701cb3d2a05d"} Oct 30 00:34:18.749508 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.749489 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:18.752044 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.752017 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" event={"ID":"d585123a-11a7-4f0f-9ae7-823a4becc3f1","Type":"ContainerStarted","Data":"2feb38d5e4474a93a8d2d0250d088a2299e16f1ec3a5ff3d9a07a6cffc353c9a"} Oct 30 00:34:18.754978 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.754951 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kcpgq" event={"ID":"af5fef70-5659-4e51-9c0e-f23dd4e998e2","Type":"ContainerStarted","Data":"7b867c82bd9207d7058b079ef01c159630610e5ff9ec93302eb67ab2a093349b"} Oct 30 00:34:18.789154 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.789092 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" podStartSLOduration=366.270458897 podStartE2EDuration="6m12.78907467s" totalImagesPullingTime="6.518615773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:18.788116353 +0000 UTC m=+38.915256195" watchObservedRunningTime="2026-10-30 00:34:18.78907467 +0000 UTC m=+38.916214514" Oct 30 00:34:18.885107 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.885029 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wzp7p" podStartSLOduration=34.365657509 podStartE2EDuration="37.88501095s" totalImagesPullingTime="3.519353441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:18.884352493 +0000 UTC m=+39.011492339" watchObservedRunningTime="2026-10-30 00:34:18.88501095 +0000 UTC m=+39.012150792" Oct 30 00:34:18.933147 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:18.933082 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kcpgq" podStartSLOduration=2.871924721 podStartE2EDuration="8.9330597s" totalImagesPullingTime="6.061134979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:18.931308136 +0000 UTC m=+39.058447979" watchObservedRunningTime="2026-10-30 00:34:18.9330597 +0000 UTC m=+39.060199543" Oct 30 00:34:19.246676 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:19.246635 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:19.246886 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:19.246765 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:19.246886 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:19.246815 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:19.246886 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:19.246881 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:21.246865301 +0000 UTC m=+41.374005134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:19.247065 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:19.246923 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:34:19.247065 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:19.246978 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:21.246960887 +0000 UTC m=+41.374100725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : configmap "kube-rbac-proxy" not found Oct 30 00:34:20.762126 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:20.762087 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" event={"ID":"d585123a-11a7-4f0f-9ae7-823a4becc3f1","Type":"ContainerStarted","Data":"b7d63e4fa41a2288ead3957f9ae295c6e8f252e79f6646b02871b3fb74f2884e"} Oct 30 00:34:20.808675 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:20.808623 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" podStartSLOduration=2.1648271279999998 podStartE2EDuration="3.808605725s" totalImagesPullingTime="1.643778597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:20.808080393 +0000 UTC m=+40.935220235" watchObservedRunningTime="2026-10-30 00:34:20.808605725 +0000 UTC m=+40.935745567" Oct 30 00:34:21.132766 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:21.132733 2584 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:21.132894 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:21.132794 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:21.133132 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:21.133119 2584 scope.go:122] "RemoveContainer" containerID="d0ce7c2a7fb68c749ef8342679ba6e2cf480d16cba39678ca62a829b5ebbf483" Oct 30 00:34:21.133332 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:21.133313 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gbrdh_openshift-console-operator(a2a1f298-60ee-4efe-8cec-88db4036a4e8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" podUID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" Oct 30 00:34:21.266005 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:21.265954 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:21.266205 ip-10-0-139-66 kubenswrapper[2584]: I1030 00:34:21.266065 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:21.266205 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:21.266069 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:21.266205 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:21.266161 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:25.266136735 +0000 UTC m=+45.393276575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:21.266205 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:21.266192 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 00:34:21.266449 ip-10-0-139-66 kubenswrapper[2584]: E1030 00:34:21.266247 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:25.266230381 +0000 UTC m=+45.393370218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : configmap "kube-rbac-proxy" not found Oct 30 00:34:21.366291 ip-10-0-139-66 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 00:34:21.384055 ip-10-0-139-66 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 00:34:21.384240 ip-10-0-139-66 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 00:34:21.384297 ip-10-0-139-66 systemd[1]: kubelet.service: Consumed 3.234s CPU time. Oct 30 00:34:21.395264 ip-10-0-139-66 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:34:21.485023 ip-10-0-139-66 kubenswrapper[4927]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:34:21.485023 ip-10-0-139-66 kubenswrapper[4927]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:34:21.485023 ip-10-0-139-66 kubenswrapper[4927]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:34:21.485023 ip-10-0-139-66 kubenswrapper[4927]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:34:21.487288 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487172 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:21.487288 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487281 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:21.487288 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487286 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:21.487288 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487289 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:21.487288 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487293 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:21.487288 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487295 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:21.487288 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487298 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487302 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487305 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487309 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487311 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487315 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487318 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487321 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487324 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487326 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487329 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487331 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487334 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487337 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487340 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487342 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487345 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487348 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487351 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:21.487557 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487354 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487356 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487359 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487362 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487365 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487368 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487371 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487374 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487377 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487379 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487382 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487385 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487387 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487391 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487394 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487397 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487399 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487402 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487405 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487407 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:21.488054 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487410 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487412 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487415 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487418 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487421 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487424 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487427 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487430 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487432 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487435 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487438 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487441 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487444 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487447 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487450 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487452 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487455 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487458 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487461 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:21.488543 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487463 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487466 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487468 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487471 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487474 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487476 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487479 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487485 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487488 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487492 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487494 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487497 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487500 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487502 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487505 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487508 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487511 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487514 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487518 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:21.489019 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487522 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487525 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487529 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487532 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487535 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487538 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487542 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487545 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487548 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487552 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487554 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487557 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487560 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487565 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487585 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487591 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487596 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487600 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487604 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487608 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:21.489486 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487612 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487616 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.487621 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488463 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488471 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488474 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488477 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488480 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488483 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488486 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488489 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488492 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488494 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488497 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488500 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488503 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488505 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488508 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488512 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:21.489995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488515 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488518 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488521 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488524 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488526 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488529 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488532 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488535 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488537 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488540 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488542 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488545 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488548 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488551 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488554 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488557 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488560 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488563 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488565 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488582 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:21.490589 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488585 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488588 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488591 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488594 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488596 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488599 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488602 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488605 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488608 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488610 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488613 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488616 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488618 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488621 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488624 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488626 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488629 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488631 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488634 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488637 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:21.491497 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488641 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488643 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488646 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488648 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488652 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488654 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488657 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488660 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488663 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488666 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488668 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488671 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488673 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488677 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488679 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488682 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488684 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488688 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488690 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:21.492051 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488693 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488696 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488698 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488701 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488705 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488709 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488713 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488717 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488719 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488722 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488725 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488727 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488730 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488733 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488735 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488738 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488741 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488748 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488752 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:21.492515 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488756 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488760 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488763 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488767 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488769 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488772 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488775 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488778 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488781 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488784 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488786 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.488789 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489282 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489288 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489290 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489293 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489296 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489300 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489302 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489305 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:21.493003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489307 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489311 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489313 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489316 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489319 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489321 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489324 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489326 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489329 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489332 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489337 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489340 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489342 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489345 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489347 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489350 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489354 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489356 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489359 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:21.493504 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489362 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489365 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489367 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489370 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489372 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489375 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489378 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489381 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489383 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489386 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489389 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489391 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489394 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489397 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489399 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489402 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489404 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489409 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489411 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489414 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:21.493999 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489417 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489419 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489422 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489426 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489429 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489431 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489434 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489437 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489439 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489444 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489447 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489449 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489452 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489455 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489457 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489460 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489463 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489465 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489468 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489471 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:21.494488 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489473 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489476 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489478 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489481 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489483 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489487 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489489 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489493 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489497 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489501 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489504 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489507 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489510 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489513 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489515 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489519 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489522 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489525 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489528 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:21.494989 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489531 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489533 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489536 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489539 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489542 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489545 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489547 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489550 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489553 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489556 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489558 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489561 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489563 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489566 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489585 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489590 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489595 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489601 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489606 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.489610 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:21.495522 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489709 4927 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489717 4927 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489725 4927 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489731 4927 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489736 4927 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489739 4927 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489743 4927 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489748 4927 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489753 4927 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489757 4927 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489761 4927 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489765 4927 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489768 4927 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489771 4927 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489774 4927 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489778 4927 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489781 4927 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489784 4927 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489789 4927 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489792 4927 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489795 4927 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489798 4927 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489802 4927 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489806 4927 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:34:21.496068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489809 4927 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489812 4927 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489816 4927 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489819 4927 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489822 4927 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489826 4927 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489829 4927 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489832 4927 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489837 4927 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489841 4927 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489845 4927 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489848 4927 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489852 4927 flags.go:64] FLAG: --enable-server="true" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489855 4927 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489859 4927 flags.go:64] FLAG: --event-burst="100" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489862 4927 flags.go:64] FLAG: --event-qps="50" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489866 4927 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489870 4927 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489873 4927 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489877 4927 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489881 4927 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489884 4927 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489887 4927 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489891 4927 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489894 4927 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:34:21.496679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489898 4927 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489901 4927 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489905 4927 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489908 4927 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489911 4927 flags.go:64] FLAG: --feature-gates="" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489915 4927 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489919 4927 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489922 4927 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489926 4927 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489929 4927 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489932 4927 flags.go:64] FLAG: --help="false" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489935 4927 flags.go:64] FLAG: --hostname-override="ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489939 4927 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489942 4927 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489945 4927 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489948 4927 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489952 4927 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489956 4927 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489959 4927 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489962 4927 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489965 4927 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489969 4927 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489972 4927 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489977 4927 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:34:21.497286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489980 4927 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489983 4927 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489986 4927 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489989 4927 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489992 4927 flags.go:64] FLAG: --lock-file="" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489995 4927 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.489998 4927 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490001 4927 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490007 4927 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490011 4927 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490014 4927 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490017 4927 flags.go:64] FLAG: --logging-format="text" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490020 4927 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490024 4927 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490027 4927 flags.go:64] FLAG: --manifest-url="" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490030 4927 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490034 4927 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490038 4927 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490042 4927 flags.go:64] FLAG: --max-pods="110" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490046 4927 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490049 4927 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490052 4927 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490055 4927 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490058 4927 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490061 4927 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:34:21.497926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490065 4927 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490072 4927 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490076 4927 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490079 4927 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490082 4927 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490085 4927 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490088 4927 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490092 4927 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490094 4927 flags.go:64] FLAG: --port="10250" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490098 4927 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490101 4927 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c56a0b4b81db9f04" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490104 4927 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490107 4927 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490110 4927 flags.go:64] FLAG: --register-node="true" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490113 4927 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490116 4927 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490120 4927 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490123 4927 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490126 4927 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490130 4927 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490133 4927 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490136 4927 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490139 4927 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490142 4927 flags.go:64] FLAG: --runonce="false" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490145 4927 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490148 4927 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490151 4927 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:34:21.498545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490154 4927 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490157 4927 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490160 4927 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490163 4927 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490166 4927 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490169 4927 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490173 4927 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490176 4927 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490179 4927 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490182 4927 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490185 4927 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490188 4927 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490194 4927 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490197 4927 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490200 4927 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490277 4927 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490283 4927 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490289 4927 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490797 4927 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490806 4927 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490813 4927 flags.go:64] FLAG: --v="2" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490822 4927 flags.go:64] FLAG: --version="false" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490831 4927 flags.go:64] FLAG: --vmodule="" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490838 4927 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.490844 4927 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:34:21.499202 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491304 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491313 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491318 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491323 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491330 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491336 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491341 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491347 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491357 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491362 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491365 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491368 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491371 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491373 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491376 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491388 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491959 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491968 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491972 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:21.499938 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491978 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491983 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491987 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491991 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491994 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.491997 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492000 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492004 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492007 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492011 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492013 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492016 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492019 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492022 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492025 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492029 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492032 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492035 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492037 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:21.500445 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492040 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492044 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492047 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492050 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492053 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492056 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492059 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492063 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492066 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492069 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492072 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492074 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492077 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492080 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492083 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492086 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492088 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492091 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492095 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492098 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:21.500937 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492101 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492104 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492106 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492109 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492117 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492120 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492124 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492129 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492132 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492136 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492139 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492144 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492147 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492151 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492154 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492157 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492159 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492164 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492167 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:21.501491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492170 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492172 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492175 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492178 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492181 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492183 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492186 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492189 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492192 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492195 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492197 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492201 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492204 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492207 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492210 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492212 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492216 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492219 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492222 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492225 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:21.501987 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492227 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492230 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492233 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492236 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492238 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492241 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492243 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492246 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492249 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.492288 4927 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492332 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492339 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492342 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492345 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492348 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492352 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:21.502549 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492355 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492357 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492361 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492364 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492366 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492369 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492372 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492374 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492377 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492380 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492383 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492385 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492388 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492392 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492395 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492398 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492400 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492403 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492406 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492409 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:21.503141 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492413 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492417 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492421 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492424 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492427 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492430 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492433 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492437 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492440 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492443 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492446 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492449 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492451 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492454 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492457 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492460 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492464 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492468 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492472 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:21.503711 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492475 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492478 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492480 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492483 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492486 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492489 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492493 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492496 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492498 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492501 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492504 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492506 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492509 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492511 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492514 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492517 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492520 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492523 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492526 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:21.504175 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492529 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492534 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492536 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492539 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492542 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492544 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492547 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492550 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492552 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492555 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492558 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492581 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492585 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492588 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492590 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492593 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492596 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492598 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492601 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492610 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:21.504802 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492612 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492615 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492618 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492621 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492624 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492626 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492629 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492632 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492635 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492637 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492640 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492643 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492645 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492649 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492653 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492656 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492659 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492661 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492664 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:21.505425 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492669 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492671 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.492674 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.497097 4927 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.497118 4927 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.497136 4927 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.497141 4927 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497203 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497208 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497212 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497215 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497218 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497221 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497224 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497227 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497230 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497233 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497236 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497240 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:21.506003 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497242 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497246 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497248 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497251 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497254 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497257 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497259 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497262 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497266 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497269 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497272 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497275 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497277 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497280 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497283 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497285 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497288 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497292 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497295 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497298 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:21.506466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497300 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497304 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497307 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497310 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497312 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497315 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497318 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497321 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497324 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497327 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497330 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497333 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497335 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497338 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497341 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497344 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497346 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497349 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497352 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497354 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:21.506991 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497358 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497361 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497364 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497367 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497369 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497372 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497375 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497379 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497382 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497386 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497390 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497393 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497396 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497399 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497401 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497406 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497410 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497413 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497416 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:21.507493 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497419 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497422 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497425 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497427 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497430 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497432 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497435 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497438 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497441 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497444 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497447 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497449 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497452 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497455 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497458 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497461 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497463 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497466 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497469 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497471 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:21.508344 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497474 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497477 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497480 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497483 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497487 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497490 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497492 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497495 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497498 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497500 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497503 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497506 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497508 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497511 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497514 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.497544 4927 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:34:21.509071 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497739 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497745 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497748 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497751 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497754 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497757 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497760 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497763 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497766 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497770 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497774 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497777 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497780 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497783 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497786 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497789 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497792 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497795 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497797 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:21.509866 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497800 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497803 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497806 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497810 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497813 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497815 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497818 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497821 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497824 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497827 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497829 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497832 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497835 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497838 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497841 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497844 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497846 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497849 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497851 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497854 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:21.510466 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497857 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497860 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497862 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497865 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497867 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497870 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497872 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497875 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497878 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497881 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497884 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497886 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497889 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497892 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497894 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497901 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497904 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497906 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497909 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497911 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:21.511168 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497914 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497917 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497921 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497925 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497928 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497931 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497935 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497938 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497940 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497943 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497946 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497949 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497952 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497955 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497957 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497960 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497963 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497965 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497968 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:21.511719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497971 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497974 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497976 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497979 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497982 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497985 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497988 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497990 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497993 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497997 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.497999 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498002 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498004 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498007 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498010 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498012 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498015 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498017 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498023 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498026 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:21.512367 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498028 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498033 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498036 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498038 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498041 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498044 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498047 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498049 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.498073 4927 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498112 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498116 4927 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498119 4927 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498122 4927 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498125 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498128 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498131 4927 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:34:21.512978 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498134 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498137 4927 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498140 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498142 4927 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498146 4927 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498150 4927 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498154 4927 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498157 4927 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498160 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498163 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498165 4927 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498168 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498171 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498173 4927 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498176 4927 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498180 4927 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498183 4927 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498190 4927 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498193 4927 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:34:21.513388 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498196 4927 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498199 4927 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498201 4927 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498204 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498207 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498210 4927 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498212 4927 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498215 4927 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498218 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498220 4927 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498223 4927 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498226 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498228 4927 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498231 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498234 4927 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498236 4927 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498239 4927 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498242 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498244 4927 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498247 4927 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:34:21.513871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498250 4927 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498253 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498256 4927 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498258 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498261 4927 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498263 4927 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498266 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498269 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498274 4927 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498276 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498281 4927 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498284 4927 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498286 4927 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498289 4927 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498292 4927 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498295 4927 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498297 4927 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498301 4927 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498305 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:34:21.514349 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498309 4927 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498312 4927 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498315 4927 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498318 4927 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498321 4927 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498324 4927 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498326 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498330 4927 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498333 4927 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498335 4927 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498338 4927 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498341 4927 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498344 4927 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498347 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498350 4927 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498353 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498355 4927 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498358 4927 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498361 4927 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498363 4927 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:34:21.514877 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498366 4927 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498370 4927 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498374 4927 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498377 4927 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498380 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498383 4927 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498386 4927 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498388 4927 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498391 4927 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498394 4927 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498396 4927 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498399 4927 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498402 4927 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498404 4927 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498408 4927 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498411 4927 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498413 4927 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498416 4927 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498419 4927 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498421 4927 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:34:21.515364 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:21.498424 4927 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.498489 4927 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.499545 4927 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.499633 4927 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.499859 4927 server.go:1014] "Starting client certificate rotation" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.500004 4927 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.500057 4927 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:28:40 +0000 UTC" deadline="2028-08-06 07:36:17.892388292 +0000 UTC" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.500085 4927 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15511h1m56.392305589s" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.500319 4927 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.501471 4927 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.502701 4927 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.504438 4927 log.go:25] "Validated CRI v1 image API" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.505185 4927 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:34:21.515977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.507150 4927 fs.go:135] Filesystem UUIDs: map[1f7dada4-297a-4354-9bba-95ad61b13ec7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7e5ce436-68db-4639-a77d-5cae854f295e:/dev/nvme0n1p3] Oct 30 00:34:21.516359 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.507179 4927 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02822fb876f7b1af92b54160650e617340f64a7fdb6739666890a6177293ed0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02822fb876f7b1af92b54160650e617340f64a7fdb6739666890a6177293ed0c/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a408f6c504d3b1bf82b07655d73f28020e68df38b6e6b2cea42f7959e75a4c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a408f6c504d3b1bf82b07655d73f28020e68df38b6e6b2cea42f7959e75a4c7/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d92191b1e485cfcff938ff26c2b3ead98855d8109dbf0604a32368ab8468df5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d92191b1e485cfcff938ff26c2b3ead98855d8109dbf0604a32368ab8468df5/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0de166975d128aded3385a28ba7be67bbbb5049da2739a6c782ec5773c353286/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0de166975d128aded3385a28ba7be67bbbb5049da2739a6c782ec5773c353286/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2feb38d5e4474a93a8d2d0250d088a2299e16f1ec3a5ff3d9a07a6cffc353c9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2feb38d5e4474a93a8d2d0250d088a2299e16f1ec3a5ff3d9a07a6cffc353c9a/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/362559db3e6cf19edbb4becaf195b300b35557ab936caddeeaa22f73ec146a34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/362559db3e6cf19edbb4becaf195b300b35557ab936caddeeaa22f73ec146a34/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51bc6520f9691e99fd90ab64a62afa1968b716593bc45b53a2552f954b2fe70d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51bc6520f9691e99fd90ab64a62afa1968b716593bc45b53a2552f954b2fe70d/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f839982ed877b3e784d19141d8d84fe90aafc67b58cd9f2fc558b9bf815186a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f839982ed877b3e784d19141d8d84fe90aafc67b58cd9f2fc558b9bf815186a/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60bdad1d49af2da468593a41842778cfb68f915e02f3a9d9687da2cfbca923a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60bdad1d49af2da468593a41842778cfb68f915e02f3a9d9687da2cfbca923a8/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6432c349df88d876c232e16662063d2336fb299a8d991324afc3580c506fd8cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6432c349df88d876c232e16662063d2336fb299a8d991324afc3580c506fd8cc/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/684da8a76cee02366fb360422152c8e343f2aee1b0af13755d7c15dd805dbb8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/684da8a76cee02366fb360422152c8e343f2aee1b0af13755d7c15dd805dbb8e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6abc4c3ed00b4ee32479a627d9d7430fc6d11552e76c7f5f95ca2497d49c12ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6abc4c3ed00b4ee32479a627d9d7430fc6d11552e76c7f5f95ca2497d49c12ca/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9845a4b8e5d215caa91fcc36fb85ede18b33a1e62172f98a640366efdd8f314c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9845a4b8e5d215caa91fcc36fb85ede18b33a1e62172f98a640366efdd8f314c/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6f6ba7f4d25e13d4b30bd1e308d93b78c0a53a27f24f5683c5127952715ba3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6f6ba7f4d25e13d4b30bd1e308d93b78c0a53a27f24f5683c5127952715ba3c/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbc6d4e5e1d7ad00156ba4c956e0e82dd18a15363c35cd46fe1a2f47340c24d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbc6d4e5e1d7ad00156ba4c956e0e82dd18a15363c35cd46fe1a2f47340c24d4/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf692c2f892512f43b57d3865d6b479a21e1ae5b94755c04ef8c8f4c7d9390c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf692c2f892512f43b57d3865d6b479a21e1ae5b94755c04ef8c8f4c7d9390c0/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c346430025e6eb00276a65082ab365066ebfcd79c5d47ec98666c235683eaf29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c346430025e6eb00276a65082ab365066ebfcd79c5d47ec98666c235683eaf29/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9c61d4151706da569456cd8d3810924fa0b567f3dfcee493090a4f690c8307f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9c61d4151706da569456cd8d3810924fa0b567f3dfcee493090a4f690c8307f/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee4611656cca54a51dfb522d4effab40913e518515f83b1ecca6f6a2eb2e1e74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee4611656cca54a51dfb522d4effab40913e518515f83b1ecca6f6a2eb2e1e74/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2cf90b0b22e22363739b21241acb788719ebe7973b5ca57e50dacb54583073c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2cf90b0b22e22363739b21241acb788719ebe7973b5ca57e50dacb54583073c/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6f2c6f0d95c8da27a44e1dd4838c6b176ba1bd571c12176bd65232f97bef791/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6f2c6f0d95c8da27a44e1dd4838c6b176ba1bd571c12176bd65232f97bef791/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb56063e9dc94da2af53ceb99cb3f0689a8b27e30c8a33d7f84144b78a6d219a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb56063e9dc94da2af53ceb99cb3f0689a8b27e30c8a33d7f84144b78a6d219a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffc949e9eb352b9cd88bef4636bb659979fc4e5bcc5441338ebaa2ee2ebfacca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffc949e9eb352b9cd88bef4636bb659979fc4e5bcc5441338ebaa2ee2ebfacca/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0203176b-a80e-4521-8bf3-833d5b973164/volumes/kubernetes.io~projected/kube-api-access-zk44v:{mountpoint:/var/lib/kubelet/pods/0203176b-a80e-4521-8bf3-833d5b973164/volumes/kubernetes.io~projected/kube-api-access-zk44v major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0432d327-4236-4bd4-8f39-d2dfe783e07e/volumes/kubernetes.io~projected/kube-api-access-9wds8:{mountpoint:/var/lib/kubelet/pods/0432d327-4236-4bd4-8f39-d2dfe783e07e/volumes/kubernetes.io~projected/kube-api-access-9wds8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/104d94b9-331d-4121-9f9f-61c191d85bba/volumes/kubernetes.io~projected/kube-api-access-7sk44:{mountpoint:/var/lib/kubelet/pods/104d94b9-331d-4121-9f9f-61c191d85bba/volumes/kubernetes.io~projected/kube-api-access-7sk44 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/104d94b9-331d-4121-9f9f-61c191d85bba/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/104d94b9-331d-4121-9f9f-61c191d85bba/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3/volumes/kubernetes.io~projected/kube-api-access-ljxqv:{mountpoint:/var/lib/kubelet/pods/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3/volumes/kubernetes.io~projected/kube-api-access-ljxqv major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5c79ec-41c6-4613-a010-af274b5f21ac/volumes/kubernetes.io~projected/kube-api-access-nh642:{mountpoint:/var/lib/kubelet/pods/1a5c79ec-41c6-4613-a010-af274b5f21ac/volumes/kubernetes.io~projected/kube-api-access-nh642 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5c79ec-41c6-4613-a010-af274b5f21ac/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1a5c79ec-41c6-4613-a010-af274b5f21ac/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c8d1f51-588d-44a5-8524-7ab17df16f1e/volumes/kubernetes.io~projected/kube-api-access-jk566:{mountpoint:/var/lib/kubelet/pods/1c8d1f51-588d-44a5-8524-7ab17df16f1e/volumes/kubernetes.io~projected/kube-api-access-jk566 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d35d272-2cbb-4667-beb8-252aec93d74c/volumes/kubernetes.io~projected/kube-api-access-sbbt7:{mountpoint:/var/lib/kubelet/pods/1d35d272-2cbb-4667-beb8-252aec93d74c/volumes/kubernetes.io~projected/kube-api-access-sbbt7 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d35d272-2cbb-4667-beb8-252aec93d74c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1d35d272-2cbb-4667-beb8-252aec93d74c/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/286e9e4c-1a89-4c7c-90b8-222e5f48af2c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/286e9e4c-1a89-4c7c-90b8-222e5f48af2c/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38d225e8-d97d-482d-9b91-33d2473d4915/volumes/kubernetes.io~projected/kube-api-access-fbtr9:{mountpoint:/var/lib/kubelet/pods/38d225e8-d97d-482d-9b91-33d2473d4915/volumes/kubernetes.io~projected/kube-api-access-fbtr9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f53e15a-0cbc-4a61-ae76-9ddc8996cabc/volumes/kubernetes.io~projected/kube-api-access-8kf5w:{mountpoint:/var/lib/kubelet/pods/3f53e15a-0cbc-4a61-ae76-9ddc8996cabc/volumes/kubernetes.io~projected/kube-api-access-8kf5w major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43af9500-17cd-44ee-9625-de42cf5c4c42/volumes/kubernetes.io~projected/kube-api-access-cjp46:{mountpoint:/var/lib/kubelet/pods/43af9500-17cd-44ee-9625-de42cf5c4c42/volumes/kubernetes.io~projected/kube-api-access-cjp46 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0/volumes/kubernetes.io~projected/kube-api-access-hdtp2:{mountpoint:/var/lib/kubelet/pods/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0/volumes/kubernetes.io~projected/kube-api-access-hdtp2 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da/volumes/kubernetes.io~projected/kube-api-access-rj9t6:{mountpoint:/var/lib/kubelet/pods/54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da/volumes/kubernetes.io~projected/kube-api-access-rj9t6 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70/volumes/kubernetes.io~projected/kube-api-access-wvqlt:{mountpoint:/var/lib/kubelet/pods/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70/volumes/kubernetes.io~projected/kube-api-access-wvqlt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62391a80-1181-4e4b-98a2-bcc1c692f460/volumes/kubernetes.io~projected/kube-api-access-h8qnl:{mountpoint:/var/lib/kubelet/pods/62391a80-1181-4e4b-98a2-bcc1c692f460/volumes/kubernetes.io~projected/kube-api-access-h8qnl major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6/volumes/kubernetes.io~projected/kube-api-access-544sc:{mountpoint:/var/lib/kubelet/pods/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6/volumes/kubernetes.io~projected/kube-api-access-544sc major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e97bcaa-c31d-45fb-913d-9f9516905bdc/volumes/kubernetes.io~projected/kube-api-access-dfcwt:{mountpoint:/var/lib/kubelet/pods/7e97bcaa-c31d-45fb-913d-9f9516905bdc/volumes/kubernetes.io~projected/kube-api-access-dfcwt major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~projected/kube-api-access-twbcq:{mountpoint:/var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~projected/kube-api-access-twbcq major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fbd9c3e-2c40-45f6-b808-76dde38075ef/volumes/kubernetes.io~projected/kube-api-access-tmbn7:{mountpoint:/var/lib/kubelet/pods/8fbd9c3e-2c40-45f6-b808-76dde38075ef/volumes/kubernetes.io~projected/kube-api-access-tmbn7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2a1f298-60ee-4efe-8cec-88db4036a4e8/volumes/kubernetes.io~projected/kube-api-access-s8brp:{mountpoint:/var/lib/kubelet/pods/a2a1f298-60ee-4efe-8cec-88db4036a4e8/volumes/kubernetes.io~projected/kube-api-access-s8brp major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2a1f298-60ee-4efe-8cec-88db4036a4e8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a2a1f298-60ee-4efe-8cec-88db4036a4e8/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af5fef70-5659-4e51-9c0e-f23dd4e998e2/volumes/kubernetes.io~projected/kube-api-access-5b4kv:{mountpoint:/var/lib/kubelet/pods/af5fef70-5659-4e51-9c0e-f23dd4e998e2/volumes/kubernetes.io~projected/kube-api-access-5b4kv major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af5fef70-5659-4e51-9c0e-f23dd4e998e2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/af5fef70-5659-4e51-9c0e-f23dd4e998e2/volumes/kubernetes.io~secret/cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~projected/kube-api-access-djt7s:{mountpoint:/var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~projected/kube-api-access-djt7s major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7b32a95-e7a4-4ba3-8735-5fb574e470ee/volumes/kubernetes.io~projected/kube-api-access-54j2g:{mountpoint:/var/lib/kubelet/pods/c7b32a95-e7a4-4ba3-8735-5fb574e470ee/volumes/kubernetes.io~projected/kube-api-access-54j2g major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cae5e6bb-b286-46da-81b9-be3d7ae18631/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/cae5e6bb-b286-46da-81b9-be3d7ae18631/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cae5e6bb-b286-46da-81b9-be3d7ae18631/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/cae5e6bb-b286-46da-81b9-be3d7ae18631/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cae5e6bb-b286-46da-81b9-be3d7ae18631/volumes/kubernetes.io~projected/kube-api-access-rjt6f:{mountpoint:/var/lib/kubelet/pods/cae5e6bb-b286-46da-81b9-be3d7ae18631/volumes/kubernetes.io~projected/kube-api-access-rjt6f major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2685357-831c-41ea-bc51-f86b156f7eb9/volumes/kubernetes.io~projected/kube-api-access-d2q99:{mountpoint:/var/lib/kubelet/pods/d2685357-831c-41ea-bc51-f86b156f7eb9/volumes/kubernetes.io~projected/kube-api-access-d2q99 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d585123a-11a7-4f0f-9ae7-823a4becc3f1/volumes/kubernetes.io~projected/kube-api-access-jljjz:{mountpoint:/var/lib/kubelet/pods/d585123a-11a7-4f0f-9ae7-823a4becc3f1/volumes/kubernetes.io~projected/kube-api-access-jljjz major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ded47351-6c07-4d2d-a5bc-20dbe69a25ab/volumes/kubernetes.io~projected/kube-api-access-8mhd7:{mountpoint:/var/lib/kubelet/pods/ded47351-6c07-4d2d-a5bc-20dbe69a25ab/volumes/kubernetes.io~projected/kube-api-access-8mhd7 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ded47351-6c07-4d2d-a5bc-20dbe69a25ab/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/ded47351-6c07-4d2d-a5bc-20dbe69a25ab/volumes/kubernetes.io~secret/default-certificate major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ded47351-6c07-4d2d-a5bc-20dbe69a25ab/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/ded47351-6c07-4d2d-a5bc-20dbe69a25ab/volumes/kubernetes.io~secret/stats-auth major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb15cf1a-dd0d-41d8-b34e-e25029799f9f/volumes/kubernetes.io~projected/kube-api-access-d5dbm:{mountpoint:/var/lib/kubelet/pods/fb15cf1a-dd0d-41d8-b34e-e25029799f9f/volumes/kubernetes.io~projected/kube-api-access-d5dbm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd212437-e8f3-4212-befe-c217498b0923/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fd212437-e8f3-4212-befe-c217498b0923/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd212437-e8f3-4212-befe-c217498b0923/volumes/kubernetes.io~projected/kube-api-access-q5gft:{mountpoint:/var/lib/kubelet/pods/fd212437-e8f3-4212-befe-c217498b0923/volumes/kubernetes.io~projected/kube-api-access-q5gft major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd212437-e8f3-4212-befe-c217498b0923/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fd212437-e8f3-4212-befe-c217498b0923/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f7161c5db3da1f47035aee71ef3f47bb10842c84d758a8a9318c9c6a1eab7298/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/494d5592b3b15054a6689df97be8de7fd1ea08b16cae1cd34c188de627a15cd0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/aad9613d24a3dbec747e08e443fa18cf22acf0a2066bed28d19bc3b3d0c80e69/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/6e1a1dd5b74169d2c76dee8ff9243519246455744e72e1306ddf547bb11d1915/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2df59f74e8ab5178bdc7227877cf3380df05732c3eabfa80c42efb5142375fd4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/38eba60098005d9cb1ee69494995d7d6d38a476669b8e6f76b0862f91bb43e33/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bf223de741b096930ffd6893a782707f5f25c8084a5b0128945a05dd9bac3af4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a16667fbcf96c29906535db29362686a8950c9d835834ce83ac52751b86e3f88/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/90abbb18ed0d2b90b1ceaca17449354ce8c18a30ed3b04f5fd4f9b5b3702c158/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/fc9e503b0ad8a774c79896ab5fbc0529c69217a3b9e6e9f2165b1dfdeee9f22b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/8debf4e99d49bac232cbc050dc3165ec2214c1cbb5ac7e0575dfc3458f0bb409/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2081d44ad5b850bcdfb046aade88cf3e16ff1f5c8167e3b413c32179f79333da/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/530ef25d1d89c25b8a8505849b75a476bbaa17e6fd829d5c02fde1e91c2db1ba/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/faea55ec0f27635237143e247bc99d2a48c501ade1b625348fd86fee108be05e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/20ccd123c548f6dbf2f4ed434f50d22b87cbdbb85aec7e7ee6b15e44ab06e882/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/20c4e82a1b8c138677b8046c62c80eeada1d82766c3551700b9485e25ddae84d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6e4873f17cbb23523ed9d562d71197f7644f2f28e2fa77a553e7387fa2778cf2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/206dda4079626a84aea46e7405f31eb4739782fe77451fffc7aaf65593737b56/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8081b66ee6a5fb6a6b2e345a21b8423f4e87ea0a016fc432f78845367c49ff01/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f7021746f87642771b058ce0aad2cbbb4632caeabf29d8fc9b9612f3a11bdd98/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8eb582ce2fdb110aa1f398f7b4cda8e6f2a173cd12308836fe15db2cb405a1ab/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/32b6291ef6ee26dfdf09391c2437b5f9aaac3bf4d6718c4f4efd796ed61e0aae/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/cd097a533db2d0a9c682a528026e9f0c4e0a95ce5502fb6e8c44501674bfc039/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/92645041a163753f54a806d73c2ca16507a81fd0e6a685baeced785181f7789d/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/918c8df51fa2d71166b523e95db82059223ec377de7d0f49e5f0a6ff64928eed/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/afbe720aba777c5cc1272063e4c558448652fd3f37da0d0f01baa631549fe1a7/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d4e5e3346925eb45dbfc80db9fba2381abb1ef46cd670f0bb54e3499f153dcb4/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/42d0b1e12369e0eb963e51dd48aefa28a5b85a6887181d47f4ddae48a0277baa/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/624c38f4e9c3a320ef63216cc248c3aa4331108f1b244d3d961960cb85eb0a06/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ffd27081365d351b91ebc156d4e0d10f30709629af5c82035557ebff1e3f7aee/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/46c879e8859efb2085ccabe4bf23bbbd90234172e585aa20375bd8171c5e0c51/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/717cf00bc7961ddcdbfb2619a3cdccb8466ff6c650aa36a7001a028c6fd242fc/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/310050f5e0668deca5fdeb432bdbbe8d3efc527c9bda23e86f3848cfbe09e2c5/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/162d93d2bbba73a4c03d0f09d6dc9aee64765cdeec03f77d548b9001ec7f7c3e/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/cf58ba52004537dfbd546c4248c747bf325136b2e8a032c59f7f74590903f168/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/16021359bbf17f9bffdde5adaa16f70a05838b5b776a255bcc8682f1199054f2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/afd79020af3bca22007086bc79cf2716caca853d31cb4f96309cf2dad4907411/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/509ffa6d2968faa44a057b66467c20bef922afd12d183dc12319b4b3a8641be5/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f7a0f77064e6d30355e6cce31b86b7dd970d75040fa089d4957c7f43f2a9d80e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/db27f798e9c538c838879e53c75aa5761a23956b406a012bbd11a4f90856fc4b/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/a43a80db02c4c0e746a4d3762466d4417f598b75cf7a896a3c2e4a75c594ad43/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/a047f7f5e3f6a52a22c0eb5a60124ba8802f67ebb5341e510f4991231ea02a5f/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/5eab2c8283e11e0a3b684c8fcca09a320e492d11a18de611073e7a7f9cefa119/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/a5d0888a2b8b8a0690fc83102bf7ab94199609822456d0f19cba4d7411a421ba/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/fef1721f8f23ad5fc0414f319c411d27b02d1d5d81098486edd32061a2f4114f/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/5275b5a357ee6db82378a8a0a7724af39b1da5b17d4843b4bba9c7057d1cbcd2/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ee4ebc0a0890a5a8c1d974db1e728f0b0e6614f30da412dd31a165d0c556f8c2/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c04b40ce3f673e664dad28d8752d9acffffe90b366adcf7e24b15a0595c7c290/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b43748b98f6a866cc12b3fc9aa4c328893058ece2a4fc5bdb8df2d0be59ce479/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8eab4de1c68d45291b6714246bbdbc3bbd0e0f1f29e56c32c7a39d5de511d29c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/2926c1f7fd85261cc1795d3dd3438be2e27e87704864bd0a2dad3f5e71bfd5f8/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/136c64794e09029b88ee70c20615c11c629048a3433e09b72447bfcf7fd1a53a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b6f92454ab377a64f70c26395ffe14379c7c5d36b8c28545b8fb9c2acd9138ce/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/e9c76f3644a2f6e0a24d7cfa2ee00a0c6279292c265d36ddddfcb3e5cd460451/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/82ea0d8580bbf3bccaa41d48b67354a3aed1159938930e1a52ae8e124fa1b2f7/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cd88847a108222e8ae14bff534d242a45d892c67d4c215bf7481c396b8319b4d/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 00:34:21.516592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.516116 4927 manager.go:217] Machine: {Timestamp:2026-10-30 00:34:21.515646138 +0000 UTC m=+0.087572352 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091667 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2036d8dc443e6c95e4d788c053e09d SystemUUID:ec2036d8-dc44-3e6c-95e4-d788c053e09d BootID:edd3eea2-a8f3-4166-bf58-4d1ae73ea320 Filesystems:[{Device:/var/lib/kubelet/pods/fb15cf1a-dd0d-41d8-b34e-e25029799f9f/volumes/kubernetes.io~projected/kube-api-access-d5dbm DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af5fef70-5659-4e51-9c0e-f23dd4e998e2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/362559db3e6cf19edbb4becaf195b300b35557ab936caddeeaa22f73ec146a34/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9c61d4151706da569456cd8d3810924fa0b567f3dfcee493090a4f690c8307f/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3/volumes/kubernetes.io~projected/kube-api-access-ljxqv DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd212437-e8f3-4212-befe-c217498b0923/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/cae5e6bb-b286-46da-81b9-be3d7ae18631/volumes/kubernetes.io~projected/kube-api-access-rjt6f DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6432c349df88d876c232e16662063d2336fb299a8d991324afc3580c506fd8cc/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d35d272-2cbb-4667-beb8-252aec93d74c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ded47351-6c07-4d2d-a5bc-20dbe69a25ab/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2feb38d5e4474a93a8d2d0250d088a2299e16f1ec3a5ff3d9a07a6cffc353c9a/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cae5e6bb-b286-46da-81b9-be3d7ae18631/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0/volumes/kubernetes.io~projected/kube-api-access-hdtp2 DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6f2c6f0d95c8da27a44e1dd4838c6b176ba1bd571c12176bd65232f97bef791/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fbd9c3e-2c40-45f6-b808-76dde38075ef/volumes/kubernetes.io~projected/kube-api-access-tmbn7 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e97bcaa-c31d-45fb-913d-9f9516905bdc/volumes/kubernetes.io~projected/kube-api-access-dfcwt DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6f6ba7f4d25e13d4b30bd1e308d93b78c0a53a27f24f5683c5127952715ba3c/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d585123a-11a7-4f0f-9ae7-823a4becc3f1/volumes/kubernetes.io~projected/kube-api-access-jljjz DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43af9500-17cd-44ee-9625-de42cf5c4c42/volumes/kubernetes.io~projected/kube-api-access-cjp46 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f839982ed877b3e784d19141d8d84fe90aafc67b58cd9f2fc558b9bf815186a/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0432d327-4236-4bd4-8f39-d2dfe783e07e/volumes/kubernetes.io~projected/kube-api-access-9wds8 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5c79ec-41c6-4613-a010-af274b5f21ac/volumes/kubernetes.io~projected/kube-api-access-nh642 DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/104d94b9-331d-4121-9f9f-61c191d85bba/volumes/kubernetes.io~projected/kube-api-access-7sk44 DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf692c2f892512f43b57d3865d6b479a21e1ae5b94755c04ef8c8f4c7d9390c0/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9845a4b8e5d215caa91fcc36fb85ede18b33a1e62172f98a640366efdd8f314c/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cae5e6bb-b286-46da-81b9-be3d7ae18631/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~projected/kube-api-access-djt7s DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5c79ec-41c6-4613-a010-af274b5f21ac/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a408f6c504d3b1bf82b07655d73f28020e68df38b6e6b2cea42f7959e75a4c7/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6abc4c3ed00b4ee32479a627d9d7430fc6d11552e76c7f5f95ca2497d49c12ca/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ded47351-6c07-4d2d-a5bc-20dbe69a25ab/volumes/kubernetes.io~projected/kube-api-access-8mhd7 DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6/volumes/kubernetes.io~projected/kube-api-access-544sc DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38d225e8-d97d-482d-9b91-33d2473d4915/volumes/kubernetes.io~projected/kube-api-access-fbtr9 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~projected/kube-api-access-twbcq DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da/volumes/kubernetes.io~projected/kube-api-access-rj9t6 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2685357-831c-41ea-bc51-f86b156f7eb9/volumes/kubernetes.io~projected/kube-api-access-d2q99 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51bc6520f9691e99fd90ab64a62afa1968b716593bc45b53a2552f954b2fe70d/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbc6d4e5e1d7ad00156ba4c956e0e82dd18a15363c35cd46fe1a2f47340c24d4/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd212437-e8f3-4212-befe-c217498b0923/volumes/kubernetes.io~projected/kube-api-access-q5gft DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2a1f298-60ee-4efe-8cec-88db4036a4e8/volumes/kubernetes.io~projected/kube-api-access-s8brp DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d92191b1e485cfcff938ff26c2b3ead98855d8109dbf0604a32368ab8468df5/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffc949e9eb352b9cd88bef4636bb659979fc4e5bcc5441338ebaa2ee2ebfacca/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3f53e15a-0cbc-4a61-ae76-9ddc8996cabc/volumes/kubernetes.io~projected/kube-api-access-8kf5w DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/62391a80-1181-4e4b-98a2-bcc1c692f460/volumes/kubernetes.io~projected/kube-api-access-h8qnl DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7b32a95-e7a4-4ba3-8735-5fb574e470ee/volumes/kubernetes.io~projected/kube-api-access-54j2g DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d35d272-2cbb-4667-beb8-252aec93d74c/volumes/kubernetes.io~projected/kube-api-access-sbbt7 DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee4611656cca54a51dfb522d4effab40913e518515f83b1ecca6f6a2eb2e1e74/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb56063e9dc94da2af53ceb99cb3f0689a8b27e30c8a33d7f84144b78a6d219a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70/volumes/kubernetes.io~projected/kube-api-access-wvqlt DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/104d94b9-331d-4121-9f9f-61c191d85bba/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd212437-e8f3-4212-befe-c217498b0923/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ded47351-6c07-4d2d-a5bc-20dbe69a25ab/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02822fb876f7b1af92b54160650e617340f64a7fdb6739666890a6177293ed0c/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af5fef70-5659-4e51-9c0e-f23dd4e998e2/volumes/kubernetes.io~projected/kube-api-access-5b4kv DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2cf90b0b22e22363739b21241acb788719ebe7973b5ca57e50dacb54583073c/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/286e9e4c-1a89-4c7c-90b8-222e5f48af2c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2a1f298-60ee-4efe-8cec-88db4036a4e8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0de166975d128aded3385a28ba7be67bbbb5049da2739a6c782ec5773c353286/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/684da8a76cee02366fb360422152c8e343f2aee1b0af13755d7c15dd805dbb8e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0203176b-a80e-4521-8bf3-833d5b973164/volumes/kubernetes.io~projected/kube-api-access-zk44v DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c8d1f51-588d-44a5-8524-7ab17df16f1e/volumes/kubernetes.io~projected/kube-api-access-jk566 DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c346430025e6eb00276a65082ab365066ebfcd79c5d47ec98666c235683eaf29/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60bdad1d49af2da468593a41842778cfb68f915e02f3a9d9687da2cfbca923a8/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02822fb876f7b1a MacAddress:5e:3b:9a:24:25:2b Speed:10000 Mtu:8901} {Name:0a408f6c504d3b1 MacAddress:32:1a:8f:18:d3:2c Speed:10000 Mtu:8901} {Name:0de166975d128ad MacAddress:5e:11:b6:81:ff:fd Speed:10000 Mtu:8901} {Name:2feb38d5e4474a9 MacAddress:3a:2f:19:b6:1e:ed Speed:10000 Mtu:8901} {Name:362559db3e6cf19 MacAddress:66:39:4f:2f:39:ce Speed:10000 Mtu:8901} {Name:5f839982ed877b3 MacAddress:46:c3:49:82:62:43 Speed:10000 Mtu:8901} {Name:a6f6ba7f4d25e13 MacAddress:ee:74:b1:b0:58:6f Speed:10000 Mtu:8901} {Name:bbc6d4e5e1d7ad0 MacAddress:ba:93:a3:a8:c3:44 Speed:10000 Mtu:8901} {Name:bf692c2f892512f MacAddress:7e:26:d2:1f:c1:cd Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dd:e7:d9:8d Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:83:6f:cf:f7:3e Speed:0 Mtu:8901} {Name:c346430025e6eb0 MacAddress:fe:4f:48:aa:13:56 Speed:10000 Mtu:8901} {Name:d9c61d4151706da MacAddress:0a:f3:31:90:71:cb Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:dd:e7:d9:8d Speed:0 Mtu:9001} {Name:f2cf90b0b22e223 MacAddress:4a:38:5c:75:41:8b Speed:10000 Mtu:8901} {Name:f6f2c6f0d95c8da MacAddress:7e:1e:6a:e3:34:f3 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:e2:d5:a5:3b:34:44 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:eb:23:b9:7e:29 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:34:21.516592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.516478 4927 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:34:21.516592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.516536 4927 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:34:21.516977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.516920 4927 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:34:21.517127 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.516953 4927 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:34:21.517206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517130 4927 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:34:21.517206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517143 4927 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:34:21.517206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517162 4927 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:34:21.517206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517189 4927 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:34:21.517386 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517218 4927 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:34:21.517477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517463 4927 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:34:21.517631 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517620 4927 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:34:21.517706 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517671 4927 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:34:21.517706 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517693 4927 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:34:21.517794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517709 4927 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:34:21.517794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517723 4927 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:34:21.517794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.517736 4927 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:34:21.518461 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.518444 4927 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:34:21.518461 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.518459 4927 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:34:21.519120 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.519099 4927 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:34:21.521297 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521283 4927 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:34:21.521341 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521297 4927 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:34:21.521404 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521394 4927 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 00:34:21.521621 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521609 4927 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:34:21.521653 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521639 4927 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:34:21.521759 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521748 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:34:21.521807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521769 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:34:21.521807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521777 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:34:21.521807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521785 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:34:21.521807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521793 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:34:21.521807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521799 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:34:21.521807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521804 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:34:21.521807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521810 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:34:21.522070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521818 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:34:21.522070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521824 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:34:21.522070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521832 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:34:21.522070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521841 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:34:21.522070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521864 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:34:21.522070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.521870 4927 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:34:21.525420 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.525407 4927 server.go:1263] "Started kubelet" Oct 30 00:34:21.525528 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.525471 4927 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:34:21.526012 ip-10-0-139-66 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:34:21.527075 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.525831 4927 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:34:21.527075 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.526531 4927 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:34:21.527462 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.527431 4927 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:34:21.529641 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.529058 4927 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.532886 4927 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.532901 4927 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.532977 4927 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:28:42 +0000 UTC" deadline="2028-04-19 12:05:45.518290103 +0000 UTC" Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.533011 4927 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12899h31m23.985281808s" Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.533016 4927 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.533029 4927 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.533030 4927 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:21.533314 4927 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found" Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.534307 4927 factory.go:55] Registering systemd factory Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.534330 4927 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.534994 4927 factory.go:153] Registering CRI-O factory Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.535012 4927 factory.go:223] Registration of the crio container factory successfully Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.535076 4927 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.535095 4927 factory.go:103] Registering Raw factory Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.535112 4927 manager.go:1199] Started watching for new ooms in manager Oct 30 00:34:21.535626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.535510 4927 manager.go:319] Starting recovery of all containers Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536180 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="286e9e4c-1a89-4c7c-90b8-222e5f48af2c" volumeName="kubernetes.io/configmap/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-konnectivity-ca" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536234 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e97bcaa-c31d-45fb-913d-9f9516905bdc" volumeName="kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536249 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89b792bc-b5bc-4153-86dc-65f1b7187a5a" volumeName="kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536262 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af5fef70-5659-4e51-9c0e-f23dd4e998e2" volumeName="kubernetes.io/secret/af5fef70-5659-4e51-9c0e-f23dd4e998e2-cert" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536279 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ded47351-6c07-4d2d-a5bc-20dbe69a25ab" volumeName="kubernetes.io/projected/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-kube-api-access-8mhd7" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536291 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd212437-e8f3-4212-befe-c217498b0923" volumeName="kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-script-lib" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536303 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd212437-e8f3-4212-befe-c217498b0923" volumeName="kubernetes.io/secret/fd212437-e8f3-4212-befe-c217498b0923-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536315 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43af9500-17cd-44ee-9625-de42cf5c4c42" volumeName="kubernetes.io/projected/43af9500-17cd-44ee-9625-de42cf5c4c42-kube-api-access-cjp46" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536332 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb15cf1a-dd0d-41d8-b34e-e25029799f9f" volumeName="kubernetes.io/projected/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kube-api-access-d5dbm" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536345 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd212437-e8f3-4212-befe-c217498b0923" volumeName="kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-config" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536358 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="104d94b9-331d-4121-9f9f-61c191d85bba" volumeName="kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-service-ca-bundle" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536371 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="104d94b9-331d-4121-9f9f-61c191d85bba" volumeName="kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-snapshots" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536384 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c8d1f51-588d-44a5-8524-7ab17df16f1e" volumeName="kubernetes.io/projected/1c8d1f51-588d-44a5-8524-7ab17df16f1e-kube-api-access-jk566" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536398 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af5fef70-5659-4e51-9c0e-f23dd4e998e2" volumeName="kubernetes.io/projected/af5fef70-5659-4e51-9c0e-f23dd4e998e2-kube-api-access-5b4kv" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536413 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc047e1-508a-4f05-ae89-58db39eb5bdf" volumeName="kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536425 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7b32a95-e7a4-4ba3-8735-5fb574e470ee" volumeName="kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-daemon-config" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536452 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd212437-e8f3-4212-befe-c217498b0923" volumeName="kubernetes.io/projected/fd212437-e8f3-4212-befe-c217498b0923-kube-api-access-q5gft" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536466 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="104d94b9-331d-4121-9f9f-61c191d85bba" volumeName="kubernetes.io/projected/104d94b9-331d-4121-9f9f-61c191d85bba-kube-api-access-7sk44" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536478 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f53e15a-0cbc-4a61-ae76-9ddc8996cabc" volumeName="kubernetes.io/projected/3f53e15a-0cbc-4a61-ae76-9ddc8996cabc-kube-api-access-8kf5w" seLinuxMountContext="" Oct 30 00:34:21.536485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536490 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e97bcaa-c31d-45fb-913d-9f9516905bdc" volumeName="kubernetes.io/projected/7e97bcaa-c31d-45fb-913d-9f9516905bdc-kube-api-access-dfcwt" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536504 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89b792bc-b5bc-4153-86dc-65f1b7187a5a" volumeName="kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536516 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89b792bc-b5bc-4153-86dc-65f1b7187a5a" volumeName="kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536529 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cae5e6bb-b286-46da-81b9-be3d7ae18631" volumeName="kubernetes.io/projected/cae5e6bb-b286-46da-81b9-be3d7ae18631-kube-api-access-rjt6f" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536541 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2685357-831c-41ea-bc51-f86b156f7eb9" volumeName="kubernetes.io/projected/d2685357-831c-41ea-bc51-f86b156f7eb9-kube-api-access-d2q99" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536553 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5c79ec-41c6-4613-a010-af274b5f21ac" volumeName="kubernetes.io/secret/1a5c79ec-41c6-4613-a010-af274b5f21ac-serving-cert" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536566 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d35d272-2cbb-4667-beb8-252aec93d74c" volumeName="kubernetes.io/secret/1d35d272-2cbb-4667-beb8-252aec93d74c-serving-cert" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536604 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89b792bc-b5bc-4153-86dc-65f1b7187a5a" volumeName="kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536626 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7b32a95-e7a4-4ba3-8735-5fb574e470ee" volumeName="kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cni-binary-copy" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536644 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3" volumeName="kubernetes.io/configmap/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-serviceca" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536655 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d35d272-2cbb-4667-beb8-252aec93d74c" volumeName="kubernetes.io/projected/1d35d272-2cbb-4667-beb8-252aec93d74c-kube-api-access-sbbt7" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536672 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70" volumeName="kubernetes.io/configmap/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-config-volume" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536685 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc047e1-508a-4f05-ae89-58db39eb5bdf" volumeName="kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536699 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7b32a95-e7a4-4ba3-8735-5fb574e470ee" volumeName="kubernetes.io/projected/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-kube-api-access-54j2g" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536713 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cae5e6bb-b286-46da-81b9-be3d7ae18631" volumeName="kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-tmp" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536725 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44120e07-23ab-4f55-96b6-b3e5dd3ad6a0" volumeName="kubernetes.io/projected/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-api-access-hdtp2" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536738 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da" volumeName="kubernetes.io/projected/54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da-kube-api-access-rj9t6" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536750 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62391a80-1181-4e4b-98a2-bcc1c692f460" volumeName="kubernetes.io/projected/62391a80-1181-4e4b-98a2-bcc1c692f460-kube-api-access-h8qnl" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536762 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89b792bc-b5bc-4153-86dc-65f1b7187a5a" volumeName="kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536775 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a1f298-60ee-4efe-8cec-88db4036a4e8" volumeName="kubernetes.io/projected/a2a1f298-60ee-4efe-8cec-88db4036a4e8-kube-api-access-s8brp" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536788 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d35d272-2cbb-4667-beb8-252aec93d74c" volumeName="kubernetes.io/configmap/1d35d272-2cbb-4667-beb8-252aec93d74c-config" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536803 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a1f298-60ee-4efe-8cec-88db4036a4e8" volumeName="kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-config" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536816 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a1f298-60ee-4efe-8cec-88db4036a4e8" volumeName="kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-trusted-ca" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536829 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc047e1-508a-4f05-ae89-58db39eb5bdf" volumeName="kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536840 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc047e1-508a-4f05-ae89-58db39eb5bdf" volumeName="kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536852 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2685357-831c-41ea-bc51-f86b156f7eb9" volumeName="kubernetes.io/configmap/d2685357-831c-41ea-bc51-f86b156f7eb9-iptables-alerter-script" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536863 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="104d94b9-331d-4121-9f9f-61c191d85bba" volumeName="kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-tmp" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536875 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89b792bc-b5bc-4153-86dc-65f1b7187a5a" volumeName="kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536886 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fbd9c3e-2c40-45f6-b808-76dde38075ef" volumeName="kubernetes.io/empty-dir/8fbd9c3e-2c40-45f6-b808-76dde38075ef-tmp-dir" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536900 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc047e1-508a-4f05-ae89-58db39eb5bdf" volumeName="kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536911 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd212437-e8f3-4212-befe-c217498b0923" volumeName="kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-env-overrides" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536923 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="286e9e4c-1a89-4c7c-90b8-222e5f48af2c" volumeName="kubernetes.io/secret/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-agent-certs" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536936 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70" volumeName="kubernetes.io/empty-dir/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-tmp-dir" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536948 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ded47351-6c07-4d2d-a5bc-20dbe69a25ab" volumeName="kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-stats-auth" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536962 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="104d94b9-331d-4121-9f9f-61c191d85bba" volumeName="kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536974 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38d225e8-d97d-482d-9b91-33d2473d4915" volumeName="kubernetes.io/projected/38d225e8-d97d-482d-9b91-33d2473d4915-kube-api-access-fbtr9" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.536986 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70" volumeName="kubernetes.io/projected/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-kube-api-access-wvqlt" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537028 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e97bcaa-c31d-45fb-913d-9f9516905bdc" volumeName="kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-binary-copy" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537042 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0432d327-4236-4bd4-8f39-d2dfe783e07e" volumeName="kubernetes.io/projected/0432d327-4236-4bd4-8f39-d2dfe783e07e-kube-api-access-9wds8" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537055 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="104d94b9-331d-4121-9f9f-61c191d85bba" volumeName="kubernetes.io/secret/104d94b9-331d-4121-9f9f-61c191d85bba-serving-cert" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537067 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3" volumeName="kubernetes.io/projected/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-kube-api-access-ljxqv" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537082 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c8d1f51-588d-44a5-8524-7ab17df16f1e" volumeName="kubernetes.io/configmap/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-config" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537095 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fbd9c3e-2c40-45f6-b808-76dde38075ef" volumeName="kubernetes.io/projected/8fbd9c3e-2c40-45f6-b808-76dde38075ef-kube-api-access-tmbn7" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537108 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc047e1-508a-4f05-ae89-58db39eb5bdf" volumeName="kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537120 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cae5e6bb-b286-46da-81b9-be3d7ae18631" volumeName="kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-tuned" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537134 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ded47351-6c07-4d2d-a5bc-20dbe69a25ab" volumeName="kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-default-certificate" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537147 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0203176b-a80e-4521-8bf3-833d5b973164" volumeName="kubernetes.io/projected/0203176b-a80e-4521-8bf3-833d5b973164-kube-api-access-zk44v" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537172 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e97bcaa-c31d-45fb-913d-9f9516905bdc" volumeName="kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537184 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89b792bc-b5bc-4153-86dc-65f1b7187a5a" volumeName="kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537198 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc047e1-508a-4f05-ae89-58db39eb5bdf" volumeName="kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537209 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36441a12-4446-4269-b6fd-61d82a00d242" volumeName="kubernetes.io/configmap/36441a12-4446-4269-b6fd-61d82a00d242-nginx-conf" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537223 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad24271-c6fb-4ac0-b2b5-b915db5b61ec" volumeName="kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537235 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6" volumeName="kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537248 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a1f298-60ee-4efe-8cec-88db4036a4e8" volumeName="kubernetes.io/secret/a2a1f298-60ee-4efe-8cec-88db4036a4e8-serving-cert" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537260 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d585123a-11a7-4f0f-9ae7-823a4becc3f1" volumeName="kubernetes.io/projected/d585123a-11a7-4f0f-9ae7-823a4becc3f1-kube-api-access-jljjz" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537271 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5c79ec-41c6-4613-a010-af274b5f21ac" volumeName="kubernetes.io/configmap/1a5c79ec-41c6-4613-a010-af274b5f21ac-config" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537283 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44120e07-23ab-4f55-96b6-b3e5dd3ad6a0" volumeName="kubernetes.io/empty-dir/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-data-volume" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537296 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62391a80-1181-4e4b-98a2-bcc1c692f460" volumeName="kubernetes.io/configmap/62391a80-1181-4e4b-98a2-bcc1c692f460-telemetry-config" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537309 4927 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5c79ec-41c6-4613-a010-af274b5f21ac" volumeName="kubernetes.io/projected/1a5c79ec-41c6-4613-a010-af274b5f21ac-kube-api-access-nh642" seLinuxMountContext="" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537321 4927 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:34:21.537485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.537329 4927 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:34:21.558696 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.558664 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/1.log" Oct 30 00:34:21.558893 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.558736 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/1.log" Oct 30 00:34:21.561145 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.561099 4927 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:34:21.562703 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.562676 4927 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:34:21.562703 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.562700 4927 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:34:21.562886 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.562726 4927 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:34:21.562886 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:21.562771 4927 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 00:34:21.577495 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.577462 4927 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:34:21.577703 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.577529 4927 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 00:34:21.579445 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.579416 4927 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:34:21.589983 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.589952 4927 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:34:21.590145 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.590065 4927 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:34:21.591648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.591612 4927 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="4261a9b702d6dda2215725bb24f97ea15250a36f68d6d2ad67a1c7aa6ba1a85d" exitCode=0 Oct 30 00:34:21.591648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.591649 4927 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="aca48198ddbe248903fe5479359678f1f5198c3cc0e62bbdf56d59b8b7915ddb" exitCode=0 Oct 30 00:34:21.591804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.591657 4927 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="6e030d222943f2c37714c9288572740eb794dff5077defc303e549f8e970c0cf" exitCode=0 Oct 30 00:34:21.591804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.591664 4927 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="ffa74123a4520de509180c8dc63c626b442a61903e7dfc0839dcad7bfb743c18" exitCode=0 Oct 30 00:34:21.591804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.591673 4927 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="63e7207ac23b170ab2e432103f6c222c995f3eb8c6e579f98abe66a906827ed4" exitCode=0 Oct 30 00:34:21.591804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.591680 4927 generic.go:356] "Generic (PLEG): container finished" podID="7e97bcaa-c31d-45fb-913d-9f9516905bdc" containerID="f82f5e2962d3ed8ed7615a5d52d65b1d94a28cefc8bd418a3f27c3783b8eb177" exitCode=0 Oct 30 00:34:21.598226 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.598192 4927 generic.go:356] "Generic (PLEG): container finished" podID="b49830c06cd5e40b61be21088e0ddfb5" containerID="f0ac33b7f52ce594dfea950a4c93120cf4c006d1b9d49ce757e379b7bf091c91" exitCode=0 Oct 30 00:34:21.619748 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.619671 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/1.log" Oct 30 00:34:21.619748 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.619700 4927 generic.go:356] "Generic (PLEG): container finished" podID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" containerID="d0ce7c2a7fb68c749ef8342679ba6e2cf480d16cba39678ca62a829b5ebbf483" exitCode=255 Oct 30 00:34:21.631803 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.631775 4927 manager.go:324] Recovery completed Oct 30 00:34:21.653989 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.653955 4927 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:34:21.653989 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.653977 4927 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:34:21.654206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654005 4927 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:34:21.654206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654165 4927 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 00:34:21.654206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654182 4927 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 00:34:21.654206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654207 4927 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 00:34:21.654411 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654215 4927 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 00:34:21.654411 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654222 4927 policy_none.go:50] "Start" Oct 30 00:34:21.654411 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654240 4927 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:34:21.654411 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654254 4927 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:34:21.654592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654439 4927 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 00:34:21.654592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654452 4927 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 00:34:21.654592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.654460 4927 policy_none.go:44] "Start" Oct 30 00:34:21.658992 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.658957 4927 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:34:21.661625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.661612 4927 manager.go:341] "Starting Device Plugin manager" Oct 30 00:34:21.662192 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:21.661642 4927 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:34:21.662226 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.662198 4927 server.go:87] "Starting device plugin registration server" Oct 30 00:34:21.662471 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.662462 4927 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:34:21.662505 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.662475 4927 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:34:21.662593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.662560 4927 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:34:21.662715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.662684 4927 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:34:21.662789 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.662725 4927 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:34:21.662789 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.662731 4927 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:34:21.662885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.662811 4927 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"] Oct 30 00:34:21.662885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.662768 4927 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:34:21.662712683Z","UUID":"6bf079a1-97bf-4939-a2e9-d0eddef995e0","Handler":null,"Name":"","Endpoint":""} Oct 30 00:34:21.663524 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.663486 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" event={"ID":"b49830c06cd5e40b61be21088e0ddfb5","Type":"ContainerStarted","Data":"7c543ba374efefa9f41f25210438dd2985c3c80839da47c273ee2b34bfa26ef0"} Oct 30 00:34:21.663524 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.663532 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" event={"ID":"b49830c06cd5e40b61be21088e0ddfb5","Type":"ContainerDied","Data":"f0ac33b7f52ce594dfea950a4c93120cf4c006d1b9d49ce757e379b7bf091c91"} Oct 30 00:34:21.663710 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.663550 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" event={"ID":"b49830c06cd5e40b61be21088e0ddfb5","Type":"ContainerStarted","Data":"ee4611656cca54a51dfb522d4effab40913e518515f83b1ecca6f6a2eb2e1e74"} Oct 30 00:34:21.663710 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.663590 4927 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.663710 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.663695 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" event={"ID":"07ffc31fbc22d538689cf3a7345b3440","Type":"ContainerStarted","Data":"fce96918bbeb5c3f32c4953e16cc3e9c417eba8500ddb1e26addefe0f727d273"} Oct 30 00:34:21.663857 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.663719 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" event={"ID":"07ffc31fbc22d538689cf3a7345b3440","Type":"ContainerStarted","Data":"9845a4b8e5d215caa91fcc36fb85ede18b33a1e62172f98a640366efdd8f314c"} Oct 30 00:34:21.663857 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.663734 4927 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.664600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.664558 4927 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:34:21.664703 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.664624 4927 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:34:21.664874 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:21.664848 4927 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:34:21.686952 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.686920 4927 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:34:21.687137 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:21.686991 4927 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.687400 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.687378 4927 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:34:21.687461 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:21.687436 4927 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.763079 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.763038 4927 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:34:21.764384 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.764364 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:34:21.764496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.764398 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:34:21.764496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.764412 4927 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:34:21.764496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.764459 4927 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.778002 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.777971 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.778070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.778007 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07ffc31fbc22d538689cf3a7345b3440-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" (UID: \"07ffc31fbc22d538689cf3a7345b3440\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.778070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.778027 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.796586 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.796549 4927 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.796753 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.796671 4927 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.878695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.878657 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07ffc31fbc22d538689cf3a7345b3440-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" (UID: \"07ffc31fbc22d538689cf3a7345b3440\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.878695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.878698 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.878875 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.878715 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.878875 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.878732 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07ffc31fbc22d538689cf3a7345b3440-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" (UID: \"07ffc31fbc22d538689cf3a7345b3440\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.878875 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.878758 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:21.878875 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:21.878793 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b49830c06cd5e40b61be21088e0ddfb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"b49830c06cd5e40b61be21088e0ddfb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:22.518446 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.518404 4927 apiserver.go:52] "Watching apiserver" Oct 30 00:34:22.568062 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.567986 4927 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:34:22.570000 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.569949 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xkfl8","kube-system/global-pull-secret-syncer-2rkgf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq","openshift-multus/multus-additional-cni-plugins-6485j","openshift-image-registry/image-registry-77dbcdb46-8kvk8","openshift-network-console/networking-console-plugin-86c478bb88-6d46m","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79","kube-system/kas-connection-checker-685895c7c9-w2hfm","openshift-multus/multus-t6nzm","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal","openshift-ingress-canary/ingress-canary-kcpgq","kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal","openshift-image-registry/image-registry-6b6dd9b44c-l6dl6","openshift-ingress/router-default-696c77d7b7-f5pjl","openshift-network-operator/iptables-alerter-s8jbb","openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb","openshift-dns/node-resolver-wdz28","openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj","openshift-insights/insights-operator-7c4986d449-tvh9g","openshift-insights/insights-runtime-extractor-9pkrm","kube-system/konnectivity-agent-xvxrc","openshift-network-diagnostics/network-check-target-wzp7p","openshift-ovn-kubernetes/ovnkube-node-fjgxn","openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj","openshift-cluster-node-tuning-operator/tuned-rhvzs","openshift-console-operator/console-operator-7cd56b8dbc-gbrdh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc","openshift-image-registry/node-ca-274sk","kube-system/kas-connection-checker-685895c7c9-kkhpn","openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl","openshift-dns/dns-default-gbrwr","openshift-multus/network-metrics-daemon-8blbw"] Oct 30 00:34:22.573869 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.573847 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:22.574139 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.574122 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:22.574871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.574856 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.575516 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.575500 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.575869 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.575855 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:22.577037 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.577021 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:22.582443 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.582427 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.583590 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.583561 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:22.584193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.584175 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.585370 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.585350 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.585447 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.585434 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.586011 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.585995 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 00:34:22.586259 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.586246 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.586333 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.586318 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 00:34:22.586384 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.586366 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 00:34:22.586429 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.586404 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.586429 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.586406 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.588390 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.588363 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 00:34:22.592085 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.592067 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 00:34:22.593179 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.593165 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 00:34:22.593946 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.593926 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 00:34:22.594033 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.593954 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.594352 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594335 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.594405 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594362 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 00:34:22.594464 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594406 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 00:34:22.594464 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594417 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.594464 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594458 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.594599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594478 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.594599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594492 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 00:34:22.594769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594753 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.594878 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594864 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 00:34:22.594934 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.594902 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.598793 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.598774 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 00:34:22.598893 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.598804 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 00:34:22.623473 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.623446 4927 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:34:22.623641 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.623546 4927 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:22.635033 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.635006 4927 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:34:22.645537 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.645512 4927 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:34:22.645666 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.645598 4927 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" Oct 30 00:34:22.646033 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.645999 4927 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:34:22.646106 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.646057 4927 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" Oct 30 00:34:22.682185 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682153 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.682342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682190 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-crio-socket\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.682342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682214 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjp46\" (UniqueName: \"kubernetes.io/projected/43af9500-17cd-44ee-9625-de42cf5c4c42-kube-api-access-cjp46\") pod \"kas-connection-checker-685895c7c9-kkhpn\" (UID: \"43af9500-17cd-44ee-9625-de42cf5c4c42\") " pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" Oct 30 00:34:22.682342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682236 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjt6f\" (UniqueName: \"kubernetes.io/projected/cae5e6bb-b286-46da-81b9-be3d7ae18631-kube-api-access-rjt6f\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.682342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682315 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-default-certificate\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.682540 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682393 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-host\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:34:22.682665 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682640 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-ovn\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.682742 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682696 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gft\" (UniqueName: \"kubernetes.io/projected/fd212437-e8f3-4212-befe-c217498b0923-kube-api-access-q5gft\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.682742 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682721 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-default-certificate\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.682871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682743 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-system-cni-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.682871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682783 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a5c79ec-41c6-4613-a010-af274b5f21ac-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:22.682871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682811 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2a1f298-60ee-4efe-8cec-88db4036a4e8-serving-cert\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:22.683024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.682874 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af5fef70-5659-4e51-9c0e-f23dd4e998e2-cert\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:22.683024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-socket-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.683122 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683023 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cnibin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.683122 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683041 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-systemd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.683122 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683046 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a5c79ec-41c6-4613-a010-af274b5f21ac-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:22.683122 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683082 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-config\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.683295 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683168 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cnibin\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.683295 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683203 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.683295 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683241 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-data-volume\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.683295 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683277 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljjz\" (UniqueName: \"kubernetes.io/projected/d585123a-11a7-4f0f-9ae7-823a4becc3f1-kube-api-access-jljjz\") pod \"migrator-66df5d496b-z7jzj\" (UID: \"d585123a-11a7-4f0f-9ae7-823a4becc3f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" Oct 30 00:34:22.683450 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683359 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-registration-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.683450 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683375 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-data-volume\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.683450 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683394 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wds8\" (UniqueName: \"kubernetes.io/projected/0432d327-4236-4bd4-8f39-d2dfe783e07e-kube-api-access-9wds8\") pod \"kas-connection-checker-685895c7c9-w2hfm\" (UID: \"0432d327-4236-4bd4-8f39-d2dfe783e07e\") " pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" Oct 30 00:34:22.683450 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683422 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbt7\" (UniqueName: \"kubernetes.io/projected/1d35d272-2cbb-4667-beb8-252aec93d74c-kube-api-access-sbbt7\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:22.683450 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683446 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-tmp-dir\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:22.683704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683471 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8brp\" (UniqueName: \"kubernetes.io/projected/a2a1f298-60ee-4efe-8cec-88db4036a4e8-kube-api-access-s8brp\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:22.683704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683499 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-system-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.683704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683523 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.683704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683530 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.683704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683588 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.683704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683616 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbn7\" (UniqueName: \"kubernetes.io/projected/8fbd9c3e-2c40-45f6-b808-76dde38075ef-kube-api-access-tmbn7\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:34:22.683704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683640 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-dbus\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:22.683704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683667 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683713 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2685357-831c-41ea-bc51-f86b156f7eb9-iptables-alerter-script\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683760 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2q99\" (UniqueName: \"kubernetes.io/projected/d2685357-831c-41ea-bc51-f86b156f7eb9-kube-api-access-d2q99\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683788 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/104d94b9-331d-4121-9f9f-61c191d85bba-serving-cert\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683804 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683822 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk566\" (UniqueName: \"kubernetes.io/projected/1c8d1f51-588d-44a5-8524-7ab17df16f1e-kube-api-access-jk566\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683851 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683876 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbcq\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683899 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-netd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683954 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/104d94b9-331d-4121-9f9f-61c191d85bba-serving-cert\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.683985 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684010 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2685357-831c-41ea-bc51-f86b156f7eb9-host-slash\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:34:22.684047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684049 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-modprobe-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684083 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684140 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-var-lib-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684197 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dbm\" (UniqueName: \"kubernetes.io/projected/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kube-api-access-d5dbm\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684234 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-kubernetes\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684271 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-tmp-dir\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684282 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-lib-modules\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684320 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-tuned\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684368 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-stats-auth\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684401 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-hostroot\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684427 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-tuned\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.684477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684432 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684473 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684543 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684589 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684617 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-slash\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684635 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684651 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9t6\" (UniqueName: \"kubernetes.io/projected/54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da-kube-api-access-rj9t6\") pod \"network-check-source-6f78d59f99-s65xl\" (UID: \"54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684675 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-sys\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684691 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cni-binary-copy\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684705 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-socket-dir-parent\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.684814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684746 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-kubelet-config\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:22.685097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684845 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:22.685097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684884 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.685097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684919 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-netns\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.685097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.684995 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd212437-e8f3-4212-befe-c217498b0923-ovn-node-metrics-cert\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.685097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685014 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-device-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.685097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685030 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-tmp\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.685097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685049 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-tmp\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.685097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685090 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685108 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-daemon-config\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685124 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685131 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cae5e6bb-b286-46da-81b9-be3d7ae18631-tmp\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685147 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685178 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-tmp\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685219 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d35d272-2cbb-4667-beb8-252aec93d74c-config\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685264 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685319 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysconfig\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685347 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.685371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685348 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685385 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685404 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbtr9\" (UniqueName: \"kubernetes.io/projected/38d225e8-d97d-482d-9b91-33d2473d4915-kube-api-access-fbtr9\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685416 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685428 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-kubelet\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685457 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-systemd-units\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685473 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d35d272-2cbb-4667-beb8-252aec93d74c-config\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685501 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk44v\" (UniqueName: \"kubernetes.io/projected/0203176b-a80e-4521-8bf3-833d5b973164-kube-api-access-zk44v\") pod \"volume-data-source-validator-58c9f5f46d-2snxj\" (UID: \"0203176b-a80e-4521-8bf3-833d5b973164\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685521 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/62391a80-1181-4e4b-98a2-bcc1c692f460-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685538 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685554 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-sys-fs\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685599 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-systemd\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685621 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhd7\" (UniqueName: \"kubernetes.io/projected/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-kube-api-access-8mhd7\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685648 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-kubelet\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685674 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh642\" (UniqueName: \"kubernetes.io/projected/1a5c79ec-41c6-4613-a010-af274b5f21ac-kube-api-access-nh642\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:22.685801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685697 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-etc-selinux\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685832 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685852 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d35d272-2cbb-4667-beb8-252aec93d74c-serving-cert\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685871 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fbd9c3e-2c40-45f6-b808-76dde38075ef-hosts-file\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685886 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-run\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685942 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-log-socket\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.685979 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-config\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686012 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686087 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-bin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686131 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d35d272-2cbb-4667-beb8-252aec93d74c-serving-cert\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686171 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxqv\" (UniqueName: \"kubernetes.io/projected/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-kube-api-access-ljxqv\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686215 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2a1f298-60ee-4efe-8cec-88db4036a4e8-config\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686211 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686239 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-node-log\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686207 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d94b9-331d-4121-9f9f-61c191d85bba-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686268 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-script-lib\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686313 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686346 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-multus\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686372 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686403 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-config-volume\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:22.686478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686472 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-bin\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686514 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-env-overrides\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686587 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686605 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686652 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8qnl\" (UniqueName: \"kubernetes.io/projected/62391a80-1181-4e4b-98a2-bcc1c692f460-kube-api-access-h8qnl\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686711 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686733 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-agent-certs\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686770 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-k8s-cni-cncf-io\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686791 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686797 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-etc-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686849 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686853 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686870 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686880 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686887 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686938 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djt7s\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686956 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4kv\" (UniqueName: \"kubernetes.io/projected/af5fef70-5659-4e51-9c0e-f23dd4e998e2-kube-api-access-5b4kv\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.686988 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd9c3e-2c40-45f6-b808-76dde38075ef-tmp-dir\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687008 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687025 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687042 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687059 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-etc-kubernetes\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687061 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd9c3e-2c40-45f6-b808-76dde38075ef-tmp-dir\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687082 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-conf\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687128 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-conf-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687180 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687217 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvqlt\" (UniqueName: \"kubernetes.io/projected/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-kube-api-access-wvqlt\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687234 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687251 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-var-lib-kubelet\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687286 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-serviceca\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687317 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687340 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36441a12-4446-4269-b6fd-61d82a00d242-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:22.687345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687361 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687376 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtp2\" (UniqueName: \"kubernetes.io/projected/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-api-access-hdtp2\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687395 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687398 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687388 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687406 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687555 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687562 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687586 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687606 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687623 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687646 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a5c79ec-41c6-4613-a010-af274b5f21ac-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687626 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687683 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-netns\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687700 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687714 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kf5w\" (UniqueName: \"kubernetes.io/projected/3f53e15a-0cbc-4a61-ae76-9ddc8996cabc-kube-api-access-8kf5w\") pod \"kas-connection-checker-685895c7c9-xkfl8\" (UID: \"3f53e15a-0cbc-4a61-ae76-9ddc8996cabc\") " pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687730 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687637 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687744 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687774 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687798 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-host\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687807 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687829 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-snapshots\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687953 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687965 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.687990 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688002 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688016 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688053 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54j2g\" (UniqueName: \"kubernetes.io/projected/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-kube-api-access-54j2g\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688065 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688074 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/104d94b9-331d-4121-9f9f-61c191d85bba-snapshots\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688084 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-multus-certs\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688113 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688112 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a5c79ec-41c6-4613-a010-af274b5f21ac-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688138 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688117 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-konnectivity-ca\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688171 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688180 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688200 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q972j\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688211 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfcwt\" (UniqueName: \"kubernetes.io/projected/7e97bcaa-c31d-45fb-913d-9f9516905bdc-kube-api-access-dfcwt\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688138 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688255 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sk44\" (UniqueName: \"kubernetes.io/projected/104d94b9-331d-4121-9f9f-61c191d85bba-kube-api-access-7sk44\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688278 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688278 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688385 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qff2g\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688395 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688428 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688430 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-th8hp\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688442 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688438 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688428 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-konnectivity-ca\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688466 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688470 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688462 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688564 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688673 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688700 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688703 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688711 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 00:34:22.688709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688753 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk5pr\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688762 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688771 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688779 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x59g5\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688801 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-smldl\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688818 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s9968\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688820 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688863 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9b4g8\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688858 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.688994 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689092 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689201 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pkqjh\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689245 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bpt2p\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689312 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-75g57\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689365 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c889x\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689414 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2pzq\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689455 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tcw96\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689589 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689599 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689674 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689703 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78zxw\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689732 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b4svw\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689758 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689781 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qpxdw\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.689889 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.690912 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:34:22.691281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.690914 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ltmbf\"" Oct 30 00:34:22.692367 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.691513 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.694844 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2a1f298-60ee-4efe-8cec-88db4036a4e8-serving-cert\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.694912 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2685357-831c-41ea-bc51-f86b156f7eb9-iptables-alerter-script\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.694912 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.694948 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af5fef70-5659-4e51-9c0e-f23dd4e998e2-cert\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.694913 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-config\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695011 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695021 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-stats-auth\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695047 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695087 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cni-binary-copy\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695109 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.695541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695509 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd212437-e8f3-4212-befe-c217498b0923-ovn-node-metrics-cert\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.695986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695643 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-daemon-config\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.695986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695775 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.695986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695904 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.695986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.695942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/62391a80-1181-4e4b-98a2-bcc1c692f460-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:22.697259 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.697243 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-env-overrides\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.697325 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.697246 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/286e9e4c-1a89-4c7c-90b8-222e5f48af2c-agent-certs\") pod \"konnectivity-agent-xvxrc\" (UID: \"286e9e4c-1a89-4c7c-90b8-222e5f48af2c\") " pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:34:22.697370 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.697357 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd212437-e8f3-4212-befe-c217498b0923-ovnkube-script-lib\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.697410 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.697383 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-config-volume\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:22.697538 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.697520 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-original-pull-secret\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:22.697695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.697680 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36441a12-4446-4269-b6fd-61d82a00d242-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:22.697756 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.697694 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.697756 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.697737 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-serviceca\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:34:22.697914 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.697898 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97bcaa-c31d-45fb-913d-9f9516905bdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.698043 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.698030 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.703453 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.703430 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljjz\" (UniqueName: \"kubernetes.io/projected/d585123a-11a7-4f0f-9ae7-823a4becc3f1-kube-api-access-jljjz\") pod \"migrator-66df5d496b-z7jzj\" (UID: \"d585123a-11a7-4f0f-9ae7-823a4becc3f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z7jzj" Oct 30 00:34:22.703453 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.703445 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjp46\" (UniqueName: \"kubernetes.io/projected/43af9500-17cd-44ee-9625-de42cf5c4c42-kube-api-access-cjp46\") pod \"kas-connection-checker-685895c7c9-kkhpn\" (UID: \"43af9500-17cd-44ee-9625-de42cf5c4c42\") " pod="kube-system/kas-connection-checker-685895c7c9-kkhpn" Oct 30 00:34:22.703611 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.703497 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjt6f\" (UniqueName: \"kubernetes.io/projected/cae5e6bb-b286-46da-81b9-be3d7ae18631-kube-api-access-rjt6f\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.704232 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.704213 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wds8\" (UniqueName: \"kubernetes.io/projected/0432d327-4236-4bd4-8f39-d2dfe783e07e-kube-api-access-9wds8\") pod \"kas-connection-checker-685895c7c9-w2hfm\" (UID: \"0432d327-4236-4bd4-8f39-d2dfe783e07e\") " pod="kube-system/kas-connection-checker-685895c7c9-w2hfm" Oct 30 00:34:22.704446 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.704430 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gft\" (UniqueName: \"kubernetes.io/projected/fd212437-e8f3-4212-befe-c217498b0923-kube-api-access-q5gft\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.704516 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.704500 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbt7\" (UniqueName: \"kubernetes.io/projected/1d35d272-2cbb-4667-beb8-252aec93d74c-kube-api-access-sbbt7\") pod \"service-ca-operator-544d749c8c-sc9sj\" (UID: \"1d35d272-2cbb-4667-beb8-252aec93d74c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" Oct 30 00:34:22.704667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.704652 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-544sc\" (UniqueName: \"kubernetes.io/projected/67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6-kube-api-access-544sc\") pod \"network-check-target-wzp7p\" (UID: \"67d4ec0d-58d5-4556-a9dc-ddb087fbd1e6\") " pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:22.705116 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.705101 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk566\" (UniqueName: \"kubernetes.io/projected/1c8d1f51-588d-44a5-8524-7ab17df16f1e-kube-api-access-jk566\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:22.705159 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.705102 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbn7\" (UniqueName: \"kubernetes.io/projected/8fbd9c3e-2c40-45f6-b808-76dde38075ef-kube-api-access-tmbn7\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:34:22.705378 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.705363 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbcq\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.705417 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.705402 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dbm\" (UniqueName: \"kubernetes.io/projected/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kube-api-access-d5dbm\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.713487 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.713459 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbtr9\" (UniqueName: \"kubernetes.io/projected/38d225e8-d97d-482d-9b91-33d2473d4915-kube-api-access-fbtr9\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:22.713817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.713798 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.713894 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.713881 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.713989 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.713974 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8brp\" (UniqueName: \"kubernetes.io/projected/a2a1f298-60ee-4efe-8cec-88db4036a4e8-kube-api-access-s8brp\") pod \"console-operator-7cd56b8dbc-gbrdh\" (UID: \"a2a1f298-60ee-4efe-8cec-88db4036a4e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:22.714099 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.714071 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhd7\" (UniqueName: \"kubernetes.io/projected/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-kube-api-access-8mhd7\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.714234 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.714218 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2q99\" (UniqueName: \"kubernetes.io/projected/d2685357-831c-41ea-bc51-f86b156f7eb9-kube-api-access-d2q99\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:34:22.716805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.716789 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54j2g\" (UniqueName: \"kubernetes.io/projected/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-kube-api-access-54j2g\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.718818 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.718792 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh642\" (UniqueName: \"kubernetes.io/projected/1a5c79ec-41c6-4613-a010-af274b5f21ac-kube-api-access-nh642\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-m5w79\" (UID: \"1a5c79ec-41c6-4613-a010-af274b5f21ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" Oct 30 00:34:22.718885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.718824 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8qnl\" (UniqueName: \"kubernetes.io/projected/62391a80-1181-4e4b-98a2-bcc1c692f460-kube-api-access-h8qnl\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:22.718885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.718862 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxqv\" (UniqueName: \"kubernetes.io/projected/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-kube-api-access-ljxqv\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:34:22.719349 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.719334 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4kv\" (UniqueName: \"kubernetes.io/projected/af5fef70-5659-4e51-9c0e-f23dd4e998e2-kube-api-access-5b4kv\") pod \"ingress-canary-kcpgq\" (UID: \"af5fef70-5659-4e51-9c0e-f23dd4e998e2\") " pod="openshift-ingress-canary/ingress-canary-kcpgq" Oct 30 00:34:22.719413 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.719399 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvqlt\" (UniqueName: \"kubernetes.io/projected/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-kube-api-access-wvqlt\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:22.719614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.719597 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtp2\" (UniqueName: \"kubernetes.io/projected/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-api-access-hdtp2\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.720109 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.720091 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfcwt\" (UniqueName: \"kubernetes.io/projected/7e97bcaa-c31d-45fb-913d-9f9516905bdc-kube-api-access-dfcwt\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.720161 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.720136 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kf5w\" (UniqueName: \"kubernetes.io/projected/3f53e15a-0cbc-4a61-ae76-9ddc8996cabc-kube-api-access-8kf5w\") pod \"kas-connection-checker-685895c7c9-xkfl8\" (UID: \"3f53e15a-0cbc-4a61-ae76-9ddc8996cabc\") " pod="kube-system/kas-connection-checker-685895c7c9-xkfl8" Oct 30 00:34:22.720331 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.720311 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9t6\" (UniqueName: \"kubernetes.io/projected/54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da-kube-api-access-rj9t6\") pod \"network-check-source-6f78d59f99-s65xl\" (UID: \"54f5b3b0-d74a-4d7e-9cf1-1fbbf65ee7da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-s65xl" Oct 30 00:34:22.720368 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.720334 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk44v\" (UniqueName: \"kubernetes.io/projected/0203176b-a80e-4521-8bf3-833d5b973164-kube-api-access-zk44v\") pod \"volume-data-source-validator-58c9f5f46d-2snxj\" (UID: \"0203176b-a80e-4521-8bf3-833d5b973164\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2snxj" Oct 30 00:34:22.720549 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.720533 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djt7s\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.722139 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.722119 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sk44\" (UniqueName: \"kubernetes.io/projected/104d94b9-331d-4121-9f9f-61c191d85bba-kube-api-access-7sk44\") pod \"insights-operator-7c4986d449-tvh9g\" (UID: \"104d94b9-331d-4121-9f9f-61c191d85bba\") " pod="openshift-insights/insights-operator-7c4986d449-tvh9g" Oct 30 00:34:22.789291 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789255 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-bin\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.789469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789300 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-k8s-cni-cncf-io\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.789469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789324 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-etc-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.789469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789332 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-bin\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.789469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789347 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:22.789469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789413 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-etc-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.789469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789436 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:22.789469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789462 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789478 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-etc-kubernetes\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789486 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-k8s-cni-cncf-io\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789492 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-conf\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789536 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-conf-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789564 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789596 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789614 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-var-lib-kubelet\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789622 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-conf-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789612 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-etc-kubernetes\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789641 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-conf\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789651 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-netns\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.789641 4927 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789680 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-host\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.789685 4927 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789692 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-var-lib-kubelet\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789683 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-netns\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789708 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-multus-certs\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789735 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-host\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789739 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-multus-certs\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.789742 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.789753 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.289727547 +0000 UTC m=+1.861653761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : secret "metrics-daemon-secret" not found Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.789832 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert podName:36441a12-4446-4269-b6fd-61d82a00d242 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.289813952 +0000 UTC m=+1.861740156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6d46m" (UID: "36441a12-4446-4269-b6fd-61d82a00d242") : secret "networking-console-plugin-cert" not found Oct 30 00:34:22.789866 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.789864 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.289852821 +0000 UTC m=+1.861779021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789885 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789928 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789952 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-crio-socket\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789978 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-host\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789989 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.789991 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790001 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-ovn\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790002 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790009 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-crio-socket\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790024 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3-host\") pod \"node-ca-274sk\" (UID: \"17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3\") " pod="openshift-image-registry/node-ca-274sk" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790033 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-system-cni-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790034 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-ovn\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790062 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-socket-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790068 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-system-cni-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790077 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cnibin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790099 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-systemd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790124 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cnibin\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790135 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-cnibin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790148 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-cnibin\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790126 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-socket-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790158 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-registration-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790164 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-run-systemd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790182 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-system-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790195 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-registration-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790197 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790235 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-system-cni-dir\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790241 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-dbus\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790261 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790283 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-netd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790307 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2685357-831c-41ea-bc51-f86b156f7eb9-host-slash\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790313 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-dbus\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790330 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-modprobe-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790343 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-cni-netd\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790347 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790355 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790378 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2685357-831c-41ea-bc51-f86b156f7eb9-host-slash\") pod \"iptables-alerter-s8jbb\" (UID: \"d2685357-831c-41ea-bc51-f86b156f7eb9\") " pod="openshift-network-operator/iptables-alerter-s8jbb" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790401 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.290386263 +0000 UTC m=+1.862312467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790431 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-var-lib-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790442 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-modprobe-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790446 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.290434803 +0000 UTC m=+1.862361022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : configmap references non-existent config key: service-ca.crt Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790466 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-var-lib-openvswitch\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790479 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-kubernetes\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790495 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-lib-modules\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790510 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-hostroot\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790561 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-kubernetes\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790566 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790601 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-lib-modules\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790616 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-slash\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790617 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-hostroot\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790650 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-sys\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790657 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-slash\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790675 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-socket-dir-parent\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790694 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-kubelet-config\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790710 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790726 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-netns\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790735 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-multus-socket-dir-parent\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790693 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-sys\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790741 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-device-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790676 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790763 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-run-netns\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790746 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ad24271-c6fb-4ac0-b2b5-b915db5b61ec-kubelet-config\") pod \"global-pull-secret-syncer-2rkgf\" (UID: \"5ad24271-c6fb-4ac0-b2b5-b915db5b61ec\") " pod="kube-system/global-pull-secret-syncer-2rkgf" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790780 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790773 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77dbcdb46-8kvk8: secret "image-registry-tls" not found Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790812 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-device-dir\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790807 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790810 4927 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790836 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790848 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls podName:afc047e1-508a-4f05-ae89-58db39eb5bdf nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.290835629 +0000 UTC m=+1.862761831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls") pod "image-registry-77dbcdb46-8kvk8" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf") : secret "image-registry-tls" not found Oct 30 00:34:22.790799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790850 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97bcaa-c31d-45fb-913d-9f9516905bdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6485j\" (UID: \"7e97bcaa-c31d-45fb-913d-9f9516905bdc\") " pod="openshift-multus/multus-additional-cni-plugins-6485j" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790870 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790874 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls podName:1c8d1f51-588d-44a5-8524-7ab17df16f1e nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.290860604 +0000 UTC m=+1.862786804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4xbzc" (UID: "1c8d1f51-588d-44a5-8524-7ab17df16f1e") : secret "samples-operator-tls" not found Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790878 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b6dd9b44c-l6dl6: secret "image-registry-tls" not found Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790897 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysconfig\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790912 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.790921 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls podName:89b792bc-b5bc-4153-86dc-65f1b7187a5a nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.290912006 +0000 UTC m=+1.862838225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls") pod "image-registry-6b6dd9b44c-l6dl6" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a") : secret "image-registry-tls" not found Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790943 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-kubelet\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790945 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysconfig\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790959 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-systemd-units\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790978 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-kubelet\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790983 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-sys-fs\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790992 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-sysctl-d\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.790999 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-systemd\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791006 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-systemd-units\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791033 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-kubelet\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791052 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-sys-fs\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791055 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-etc-selinux\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791081 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb15cf1a-dd0d-41d8-b34e-e25029799f9f-etc-selinux\") pod \"aws-ebs-csi-driver-node-shfhq\" (UID: \"fb15cf1a-dd0d-41d8-b34e-e25029799f9f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shfhq" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791063 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-etc-systemd\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791090 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791119 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fbd9c3e-2c40-45f6-b808-76dde38075ef-hosts-file\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791122 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-run-ovn-kubernetes\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791093 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-host-kubelet\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791152 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-run\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791179 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-log-socket\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791198 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-bin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791200 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cae5e6bb-b286-46da-81b9-be3d7ae18631-run\") pod \"tuned-rhvzs\" (UID: \"cae5e6bb-b286-46da-81b9-be3d7ae18631\") " pod="openshift-cluster-node-tuning-operator/tuned-rhvzs" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791202 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fbd9c3e-2c40-45f6-b808-76dde38075ef-hosts-file\") pod \"node-resolver-wdz28\" (UID: \"8fbd9c3e-2c40-45f6-b808-76dde38075ef\") " pod="openshift-dns/node-resolver-wdz28" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791231 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-bin\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791234 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-node-log\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791251 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-log-socket\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791271 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd212437-e8f3-4212-befe-c217498b0923-node-log\") pod \"ovnkube-node-fjgxn\" (UID: \"fd212437-e8f3-4212-befe-c217498b0923\") " pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.791321 4927 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791270 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791346 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-multus\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.791353 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls podName:62391a80-1181-4e4b-98a2-bcc1c692f460 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.29134473 +0000 UTC m=+1.863270930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7b7wb" (UID: "62391a80-1181-4e4b-98a2-bcc1c692f460") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791366 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7b32a95-e7a4-4ba3-8735-5fb574e470ee-host-var-lib-cni-multus\") pod \"multus-t6nzm\" (UID: \"c7b32a95-e7a4-4ba3-8735-5fb574e470ee\") " pod="openshift-multus/multus-t6nzm" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.791372 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.791434 4927 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:22.792754 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:22.791459 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:23.291452779 +0000 UTC m=+1.863378979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:22.874763 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:22.874735 4927 scope.go:122] "RemoveContainer" containerID="d0ce7c2a7fb68c749ef8342679ba6e2cf480d16cba39678ca62a829b5ebbf483" Oct 30 00:34:23.296715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.296610 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:23.296715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.296697 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.296732 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.296795 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.296807 4927 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.296827 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.296843 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.296860 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.296930 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.296891449 +0000 UTC m=+2.868817672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.296940 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.296957 4927 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.296956 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.296982 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77dbcdb46-8kvk8: secret "image-registry-tls" not found Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.296992 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.296971318 +0000 UTC m=+2.868897535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : secret "metrics-daemon-secret" not found Oct 30 00:34:23.297018 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297021 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls podName:afc047e1-508a-4f05-ae89-58db39eb5bdf nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.29701397 +0000 UTC m=+2.868940174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls") pod "image-registry-77dbcdb46-8kvk8" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf") : secret "image-registry-tls" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297048 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.297029386 +0000 UTC m=+2.868955586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297063 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert podName:36441a12-4446-4269-b6fd-61d82a00d242 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.297057536 +0000 UTC m=+2.868983736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6d46m" (UID: "36441a12-4446-4269-b6fd-61d82a00d242") : secret "networking-console-plugin-cert" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297077 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.297070421 +0000 UTC m=+2.868996621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : configmap references non-existent config key: service-ca.crt Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.297105 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.297132 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.297170 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.297188 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297239 4927 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297260 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297274 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b6dd9b44c-l6dl6: secret "image-registry-tls" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297295 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls podName:1c8d1f51-588d-44a5-8524-7ab17df16f1e nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.297284152 +0000 UTC m=+2.869210356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4xbzc" (UID: "1c8d1f51-588d-44a5-8524-7ab17df16f1e") : secret "samples-operator-tls" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297330 4927 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297331 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls podName:89b792bc-b5bc-4153-86dc-65f1b7187a5a nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.297319506 +0000 UTC m=+2.869245726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls") pod "image-registry-6b6dd9b44c-l6dl6" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a") : secret "image-registry-tls" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297264 4927 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297378 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.297364592 +0000 UTC m=+2.869290803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:23.297433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.297394 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls podName:62391a80-1181-4e4b-98a2-bcc1c692f460 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:24.297385341 +0000 UTC m=+2.869311566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7b7wb" (UID: "62391a80-1181-4e4b-98a2-bcc1c692f460") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:23.434877 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.434836 4927 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:23.632401 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.632372 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:34:23.632869 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.632848 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/1.log" Oct 30 00:34:23.632929 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.632895 4927 generic.go:356] "Generic (PLEG): container finished" podID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" containerID="4253e12306e30f6e7b534dc4141fe42e96334d60603c635a06f11d3a7baa8562" exitCode=255 Oct 30 00:34:23.633006 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.632978 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" event={"ID":"a2a1f298-60ee-4efe-8cec-88db4036a4e8","Type":"ContainerDied","Data":"4253e12306e30f6e7b534dc4141fe42e96334d60603c635a06f11d3a7baa8562"} Oct 30 00:34:23.633087 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.633071 4927 scope.go:122] "RemoveContainer" containerID="d0ce7c2a7fb68c749ef8342679ba6e2cf480d16cba39678ca62a829b5ebbf483" Oct 30 00:34:23.633291 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:23.633264 4927 scope.go:122] "RemoveContainer" containerID="4253e12306e30f6e7b534dc4141fe42e96334d60603c635a06f11d3a7baa8562" Oct 30 00:34:23.633561 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:23.633524 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gbrdh_openshift-console-operator(a2a1f298-60ee-4efe-8cec-88db4036a4e8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" podUID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" Oct 30 00:34:24.068717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.068641 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:24.307733 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.307684 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:24.307961 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.307787 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:24.307961 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.307820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:24.307961 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.307845 4927 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 00:34:24.307961 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.307932 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert podName:36441a12-4446-4269-b6fd-61d82a00d242 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.307907365 +0000 UTC m=+4.879833582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6d46m" (UID: "36441a12-4446-4269-b6fd-61d82a00d242") : secret "networking-console-plugin-cert" not found Oct 30 00:34:24.307961 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.307849 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:24.307961 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.307955 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.307945495 +0000 UTC m=+4.879871695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : configmap references non-existent config key: service-ca.crt Oct 30 00:34:24.307961 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.307936 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.308002 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308023 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.308003957 +0000 UTC m=+4.879930160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308026 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308041 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77dbcdb46-8kvk8: secret "image-registry-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.308063 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308077 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls podName:afc047e1-508a-4f05-ae89-58db39eb5bdf nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.308065738 +0000 UTC m=+4.879991962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls") pod "image-registry-77dbcdb46-8kvk8" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf") : secret "image-registry-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308077 4927 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308127 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308140 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b6dd9b44c-l6dl6: secret "image-registry-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308149 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls podName:1c8d1f51-588d-44a5-8524-7ab17df16f1e nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.308116093 +0000 UTC m=+4.880042319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4xbzc" (UID: "1c8d1f51-588d-44a5-8524-7ab17df16f1e") : secret "samples-operator-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308162 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls podName:89b792bc-b5bc-4153-86dc-65f1b7187a5a nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.308156376 +0000 UTC m=+4.880082576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls") pod "image-registry-6b6dd9b44c-l6dl6" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a") : secret "image-registry-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.308175 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.308204 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.308228 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308246 4927 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308294 4927 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308301 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:24.308292 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308301 4927 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:24.308850 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.308248 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:24.308850 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308296 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls podName:62391a80-1181-4e4b-98a2-bcc1c692f460 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.308281278 +0000 UTC m=+4.880207493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-7b7wb" (UID: "62391a80-1181-4e4b-98a2-bcc1c692f460") : secret "cluster-monitoring-operator-tls" not found Oct 30 00:34:24.308850 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308353 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls podName:44120e07-23ab-4f55-96b6-b3e5dd3ad6a0 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.308339884 +0000 UTC m=+4.880266083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pkrm" (UID: "44120e07-23ab-4f55-96b6-b3e5dd3ad6a0") : secret "insights-runtime-extractor-tls" not found Oct 30 00:34:24.308850 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308369 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.308360949 +0000 UTC m=+4.880287149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:24.308850 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.308407 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:26.30840225 +0000 UTC m=+4.880328450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : secret "metrics-daemon-secret" not found Oct 30 00:34:24.314664 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.314644 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdz28_8fbd9c3e-2c40-45f6-b808-76dde38075ef/dns-node-resolver/0.log" Oct 30 00:34:24.330386 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.330322 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:24.638037 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.638002 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:34:24.638521 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.638327 4927 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:34:24.638521 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.638339 4927 scope.go:122] "RemoveContainer" containerID="4253e12306e30f6e7b534dc4141fe42e96334d60603c635a06f11d3a7baa8562" Oct 30 00:34:24.638614 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:24.638521 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gbrdh_openshift-console-operator(a2a1f298-60ee-4efe-8cec-88db4036a4e8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" podUID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" Oct 30 00:34:24.915938 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:24.915858 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-274sk_17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3/node-ca/0.log" Oct 30 00:34:25.715313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:25.715273 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kcpgq_af5fef70-5659-4e51-9c0e-f23dd4e998e2/serve-healthcheck-canary/0.log" Oct 30 00:34:26.315834 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.315805 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-z7jzj_d585123a-11a7-4f0f-9ae7-823a4becc3f1/migrator/0.log" Oct 30 00:34:26.328445 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328416 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:26.328536 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328452 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:26.328536 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328477 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:26.328536 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328505 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:26.328536 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328524 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328599 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328602 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328622 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:30.328598706 +0000 UTC m=+8.900524907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : configmap references non-existent config key: service-ca.crt Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328653 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328663 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:30.328645485 +0000 UTC m=+8.900571700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328666 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77dbcdb46-8kvk8: secret "image-registry-tls" not found Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328687 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328694 4927 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328709 4927 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b6dd9b44c-l6dl6: secret "image-registry-tls" not found Oct 30 00:34:26.328743 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328702 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls podName:afc047e1-508a-4f05-ae89-58db39eb5bdf nodeName:}" failed. No retries permitted until 2026-10-30 00:34:30.328691201 +0000 UTC m=+8.900617416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls") pod "image-registry-77dbcdb46-8kvk8" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf") : secret "image-registry-tls" not found Oct 30 00:34:26.329171 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328753 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls podName:89b792bc-b5bc-4153-86dc-65f1b7187a5a nodeName:}" failed. No retries permitted until 2026-10-30 00:34:30.32874393 +0000 UTC m=+8.900670130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls") pod "image-registry-6b6dd9b44c-l6dl6" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a") : secret "image-registry-tls" not found Oct 30 00:34:26.329171 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328798 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:26.329171 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328825 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:26.329171 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328858 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:26.329171 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328921 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:26.329171 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328929 4927 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:26.329171 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328968 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:30.328954015 +0000 UTC m=+8.900880233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:26.329171 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:26.328986 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:30.32897697 +0000 UTC m=+8.900903174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : secret "metrics-daemon-secret" not found Oct 30 00:34:26.329171 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.328980 4927 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:34:26.331451 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.331423 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/62391a80-1181-4e4b-98a2-bcc1c692f460-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-7b7wb\" (UID: \"62391a80-1181-4e4b-98a2-bcc1c692f460\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:26.331556 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.331425 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c8d1f51-588d-44a5-8524-7ab17df16f1e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4xbzc\" (UID: \"1c8d1f51-588d-44a5-8524-7ab17df16f1e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:26.331727 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.331707 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36441a12-4446-4269-b6fd-61d82a00d242-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6d46m\" (UID: \"36441a12-4446-4269-b6fd-61d82a00d242\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:26.331770 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.331712 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44120e07-23ab-4f55-96b6-b3e5dd3ad6a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pkrm\" (UID: \"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0\") " pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:26.474993 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.474954 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" Oct 30 00:34:26.474993 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.474986 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" Oct 30 00:34:26.477121 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.477086 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" Oct 30 00:34:26.485170 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.485144 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9pkrm" Oct 30 00:34:26.519600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.517831 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-m5w79_1a5c79ec-41c6-4613-a010-af274b5f21ac/kube-storage-version-migrator-operator/0.log" Oct 30 00:34:26.819723 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.819676 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6d46m"] Oct 30 00:34:26.823165 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.823116 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc"] Oct 30 00:34:26.823845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.823814 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9pkrm"] Oct 30 00:34:26.824021 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:26.823997 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36441a12_4446_4269_b6fd_61d82a00d242.slice/crio-2e9c82e218e6d02fd6bc55535c3f200425448d497b4a2beab937a846e7928d35 WatchSource:0}: Error finding container 2e9c82e218e6d02fd6bc55535c3f200425448d497b4a2beab937a846e7928d35: Status 404 returned error can't find the container with id 2e9c82e218e6d02fd6bc55535c3f200425448d497b4a2beab937a846e7928d35 Oct 30 00:34:26.826205 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.826187 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:34:26.834087 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:26.834064 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb"] Oct 30 00:34:26.843927 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:26.843897 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62391a80_1181_4e4b_98a2_bcc1c692f460.slice/crio-b551642a7992559be350face84682f5cdc7516e7aea7b4c098d87f1bc861e351 WatchSource:0}: Error finding container b551642a7992559be350face84682f5cdc7516e7aea7b4c098d87f1bc861e351: Status 404 returned error can't find the container with id b551642a7992559be350face84682f5cdc7516e7aea7b4c098d87f1bc861e351 Oct 30 00:34:27.656794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:27.656730 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" event={"ID":"36441a12-4446-4269-b6fd-61d82a00d242","Type":"ContainerStarted","Data":"2e9c82e218e6d02fd6bc55535c3f200425448d497b4a2beab937a846e7928d35"} Oct 30 00:34:27.659052 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:27.659012 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" event={"ID":"62391a80-1181-4e4b-98a2-bcc1c692f460","Type":"ContainerStarted","Data":"b551642a7992559be350face84682f5cdc7516e7aea7b4c098d87f1bc861e351"} Oct 30 00:34:27.660698 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:27.660672 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" event={"ID":"1c8d1f51-588d-44a5-8524-7ab17df16f1e","Type":"ContainerStarted","Data":"c6311e05618e6811eb42543db981898bfabd7451c45ff42cbf99617008dfb9e2"} Oct 30 00:34:27.663682 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:27.663652 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pkrm" event={"ID":"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0","Type":"ContainerStarted","Data":"7b0ddcc1b18443749856f329ec1227b9b1b73dbf4f0d0a8334e8053ee78ac3f9"} Oct 30 00:34:27.663779 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:27.663717 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pkrm" event={"ID":"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0","Type":"ContainerStarted","Data":"674a8ba42670258db207a9250c7c7eecfdadd5d023a6cba960ec36b0f4e9c848"} Oct 30 00:34:28.671222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:28.671175 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pkrm" event={"ID":"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0","Type":"ContainerStarted","Data":"15d02d9fee76685f61cf158d0f5cf9037c1ca9058377e57418acd1a378cfc9f9"} Oct 30 00:34:28.672918 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:28.672887 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6d46m" event={"ID":"36441a12-4446-4269-b6fd-61d82a00d242","Type":"ContainerStarted","Data":"ac00389cfefb1598a3756044b1dab433d79b09efaf4d39485a488468504b2d7d"} Oct 30 00:34:29.678245 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:29.678205 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pkrm" event={"ID":"44120e07-23ab-4f55-96b6-b3e5dd3ad6a0","Type":"ContainerStarted","Data":"d30c5da82e224213ba15f467157fab06147c461fa26558a55ad3ab96ebfb5019"} Oct 30 00:34:29.679790 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:29.679764 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-7b7wb" event={"ID":"62391a80-1181-4e4b-98a2-bcc1c692f460","Type":"ContainerStarted","Data":"9ddbe842c0189f4c62f98319e420f37e32b29e23eedded21ca9b9f3323d6b49c"} Oct 30 00:34:29.681462 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:29.681434 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" event={"ID":"1c8d1f51-588d-44a5-8524-7ab17df16f1e","Type":"ContainerStarted","Data":"6c0a2449269d09f29154a0d94ed33844f427f9730c93d31408d8bb55bdb4c316"} Oct 30 00:34:29.681591 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:29.681486 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4xbzc" event={"ID":"1c8d1f51-588d-44a5-8524-7ab17df16f1e","Type":"ContainerStarted","Data":"af547cf7220315606eef7e206cbd475e69c05509a3d74dce8d2fafd35ee107d1"} Oct 30 00:34:30.368162 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.368098 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:30.368162 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.368168 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:30.368460 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.368190 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:30.368460 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.368217 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:30.368460 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.368256 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:30.368460 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:30.368267 4927 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 00:34:30.368460 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:30.368344 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs podName:ded47351-6c07-4d2d-a5bc-20dbe69a25ab nodeName:}" failed. No retries permitted until 2026-10-30 00:34:38.368327725 +0000 UTC m=+16.940253925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs") pod "router-default-696c77d7b7-f5pjl" (UID: "ded47351-6c07-4d2d-a5bc-20dbe69a25ab") : secret "router-metrics-certs-default" not found Oct 30 00:34:30.368460 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:30.368353 4927 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 00:34:30.368460 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:30.368413 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs podName:38d225e8-d97d-482d-9b91-33d2473d4915 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:38.368397417 +0000 UTC m=+16.940323632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs") pod "network-metrics-daemon-8blbw" (UID: "38d225e8-d97d-482d-9b91-33d2473d4915") : secret "metrics-daemon-secret" not found Oct 30 00:34:30.368460 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.368284 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:30.368861 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:30.368501 4927 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 00:34:30.368861 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:30.368585 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls podName:59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:38.368552401 +0000 UTC m=+16.940478615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls") pod "dns-default-gbrwr" (UID: "59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70") : secret "dns-default-metrics-tls" not found Oct 30 00:34:30.368995 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.368970 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-service-ca-bundle\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:30.370784 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.370758 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"image-registry-6b6dd9b44c-l6dl6\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:30.370899 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.370809 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"image-registry-77dbcdb46-8kvk8\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:30.376809 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.376782 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:30.383019 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.382989 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:30.515083 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.515053 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fjgxn" Oct 30 00:34:30.706522 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.706487 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b6dd9b44c-l6dl6"] Oct 30 00:34:30.709200 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:30.709173 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77dbcdb46-8kvk8"] Oct 30 00:34:30.710882 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:30.710859 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b792bc_b5bc_4153_86dc_65f1b7187a5a.slice/crio-659a6e1daa8b4e933270b45c78b0d1bfc28b293d12c9614733a5346173252068 WatchSource:0}: Error finding container 659a6e1daa8b4e933270b45c78b0d1bfc28b293d12c9614733a5346173252068: Status 404 returned error can't find the container with id 659a6e1daa8b4e933270b45c78b0d1bfc28b293d12c9614733a5346173252068 Oct 30 00:34:30.712936 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:30.712875 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc047e1_508a_4f05_ae89_58db39eb5bdf.slice/crio-7cedda7c0a487d02309cf2191e1507ec9eb2092817dba562a56bc0f3e85a9802 WatchSource:0}: Error finding container 7cedda7c0a487d02309cf2191e1507ec9eb2092817dba562a56bc0f3e85a9802: Status 404 returned error can't find the container with id 7cedda7c0a487d02309cf2191e1507ec9eb2092817dba562a56bc0f3e85a9802 Oct 30 00:34:31.050557 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:31.050468 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:31.051125 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:31.051097 4927 scope.go:122] "RemoveContainer" containerID="4253e12306e30f6e7b534dc4141fe42e96334d60603c635a06f11d3a7baa8562" Oct 30 00:34:31.051467 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:31.051422 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gbrdh_openshift-console-operator(a2a1f298-60ee-4efe-8cec-88db4036a4e8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" podUID="a2a1f298-60ee-4efe-8cec-88db4036a4e8" Oct 30 00:34:31.689274 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:31.689225 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" event={"ID":"89b792bc-b5bc-4153-86dc-65f1b7187a5a","Type":"ContainerStarted","Data":"363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8"} Oct 30 00:34:31.689457 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:31.689293 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" event={"ID":"89b792bc-b5bc-4153-86dc-65f1b7187a5a","Type":"ContainerStarted","Data":"659a6e1daa8b4e933270b45c78b0d1bfc28b293d12c9614733a5346173252068"} Oct 30 00:34:31.690807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:31.690779 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" event={"ID":"afc047e1-508a-4f05-ae89-58db39eb5bdf","Type":"ContainerStarted","Data":"7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb"} Oct 30 00:34:31.690934 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:31.690818 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" event={"ID":"afc047e1-508a-4f05-ae89-58db39eb5bdf","Type":"ContainerStarted","Data":"7cedda7c0a487d02309cf2191e1507ec9eb2092817dba562a56bc0f3e85a9802"} Oct 30 00:34:31.993429 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:31.993341 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:32.693819 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:32.693786 4927 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:34:32.693979 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:32.693844 4927 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:34:33.435740 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:33.435704 4927 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:33.436141 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:33.436071 4927 scope.go:122] "RemoveContainer" containerID="4253e12306e30f6e7b534dc4141fe42e96334d60603c635a06f11d3a7baa8562" Oct 30 00:34:33.699461 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:33.699381 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:34:33.699610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:33.699487 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" event={"ID":"a2a1f298-60ee-4efe-8cec-88db4036a4e8","Type":"ContainerStarted","Data":"efc5cca21088628fe608443823359a460eb00b7066a1954d0e40af99b8a9e88d"} Oct 30 00:34:33.699805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:33.699786 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:33.808522 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:33.808494 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gbrdh" Oct 30 00:34:34.441896 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:34:34.441859 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:34:36.399842 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:36.399805 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b6dd9b44c-l6dl6"] Oct 30 00:34:36.400281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:36.399943 4927 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:34:36.955861 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:36.955822 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:38.442822 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.442768 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:38.443384 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.442874 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:38.443384 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.442905 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:38.445242 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.445217 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d225e8-d97d-482d-9b91-33d2473d4915-metrics-certs\") pod \"network-metrics-daemon-8blbw\" (UID: \"38d225e8-d97d-482d-9b91-33d2473d4915\") " pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:38.445383 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.445363 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70-metrics-tls\") pod \"dns-default-gbrwr\" (UID: \"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70\") " pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:38.445436 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.445360 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded47351-6c07-4d2d-a5bc-20dbe69a25ab-metrics-certs\") pod \"router-default-696c77d7b7-f5pjl\" (UID: \"ded47351-6c07-4d2d-a5bc-20dbe69a25ab\") " pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:38.475369 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.475330 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:38.477447 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.477413 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8blbw" Oct 30 00:34:38.484659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.484636 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:38.754362 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.754289 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:34:38.810961 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.810924 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8blbw"] Oct 30 00:34:38.814902 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:38.814875 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d225e8_d97d_482d_9b91_33d2473d4915.slice/crio-dc669c910ced59e9eca0fd7a4ae0722f97682642d087455ef68844a24ffa2d70 WatchSource:0}: Error finding container dc669c910ced59e9eca0fd7a4ae0722f97682642d087455ef68844a24ffa2d70: Status 404 returned error can't find the container with id dc669c910ced59e9eca0fd7a4ae0722f97682642d087455ef68844a24ffa2d70 Oct 30 00:34:38.818032 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:38.818009 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fcdf40_4ce4_4d7f_ba4f_a8cc8a405c70.slice/crio-33f57b9f6c7e84d1da4f2d4c648db168b2308ca825262539a89624a4368402ad WatchSource:0}: Error finding container 33f57b9f6c7e84d1da4f2d4c648db168b2308ca825262539a89624a4368402ad: Status 404 returned error can't find the container with id 33f57b9f6c7e84d1da4f2d4c648db168b2308ca825262539a89624a4368402ad Oct 30 00:34:38.822215 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.822194 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gbrwr"] Oct 30 00:34:38.825380 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:38.825359 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-696c77d7b7-f5pjl"] Oct 30 00:34:38.828012 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:38.827987 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded47351_6c07_4d2d_a5bc_20dbe69a25ab.slice/crio-bb856f03cb918ed8d3771f2c8be2b2180d5fcad3a738eb2221b593924dc024e4 WatchSource:0}: Error finding container bb856f03cb918ed8d3771f2c8be2b2180d5fcad3a738eb2221b593924dc024e4: Status 404 returned error can't find the container with id bb856f03cb918ed8d3771f2c8be2b2180d5fcad3a738eb2221b593924dc024e4 Oct 30 00:34:39.724390 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:39.724357 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8blbw" event={"ID":"38d225e8-d97d-482d-9b91-33d2473d4915","Type":"ContainerStarted","Data":"dc669c910ced59e9eca0fd7a4ae0722f97682642d087455ef68844a24ffa2d70"} Oct 30 00:34:39.726026 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:39.725942 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbrwr" event={"ID":"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70","Type":"ContainerStarted","Data":"33f57b9f6c7e84d1da4f2d4c648db168b2308ca825262539a89624a4368402ad"} Oct 30 00:34:39.728896 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:39.728865 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-696c77d7b7-f5pjl" event={"ID":"ded47351-6c07-4d2d-a5bc-20dbe69a25ab","Type":"ContainerStarted","Data":"c2d18eefae41a2e858cbf2fd4982008421ba0b21266d95355ed9d94fb1f7a5f3"} Oct 30 00:34:39.728988 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:39.728917 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-696c77d7b7-f5pjl" event={"ID":"ded47351-6c07-4d2d-a5bc-20dbe69a25ab","Type":"ContainerStarted","Data":"bb856f03cb918ed8d3771f2c8be2b2180d5fcad3a738eb2221b593924dc024e4"} Oct 30 00:34:40.112377 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.112296 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xvxrc" Oct 30 00:34:40.476115 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.476077 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:40.479237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.479208 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:40.734553 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.734457 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbrwr" event={"ID":"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70","Type":"ContainerStarted","Data":"c6feb03ca68d57fb5c6efc94da768cc1bbe85e102019ea8b24ec3395ec4c4145"} Oct 30 00:34:40.734553 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.734531 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbrwr" event={"ID":"59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70","Type":"ContainerStarted","Data":"ab3fb29cc314b5319a2482856ccaa99b8c2b8b1d829f9d9065a1fc281bc3412b"} Oct 30 00:34:40.735043 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.734565 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:40.736371 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.736338 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8blbw" event={"ID":"38d225e8-d97d-482d-9b91-33d2473d4915","Type":"ContainerStarted","Data":"623f9c8bc88e68c98019464d1fe832ce64cee5aace25b7ecb41dfffe4eeb6882"} Oct 30 00:34:40.736504 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.736404 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8blbw" event={"ID":"38d225e8-d97d-482d-9b91-33d2473d4915","Type":"ContainerStarted","Data":"8d873029c9714edda33230ee329c45189896aa41fb0de792ae666a224a867f38"} Oct 30 00:34:40.736553 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.736527 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:40.737785 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:40.737763 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-696c77d7b7-f5pjl" Oct 30 00:34:44.784228 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.784193 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g9nnw"] Oct 30 00:34:44.822517 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.822488 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:44.855301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.855255 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:34:44.855477 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.855392 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:34:44.865346 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.865319 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:34:44.865346 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.865327 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x9vrg\"" Oct 30 00:34:44.870279 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.870204 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:34:44.901800 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.901755 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-wtmp\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:44.902015 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.901805 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:44.902015 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.901838 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-accelerators-collector-config\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:44.902015 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.901957 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a393f07-726f-49bb-925f-913d3c4c729a-metrics-client-ca\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:44.902192 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.902117 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a393f07-726f-49bb-925f-913d3c4c729a-sys\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:44.902243 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.902195 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-textfile\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:44.902440 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.902406 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-tls\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:44.902589 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.902459 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvcq\" (UniqueName: \"kubernetes.io/projected/0a393f07-726f-49bb-925f-913d3c4c729a-kube-api-access-ctvcq\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:44.902589 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:44.902489 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a393f07-726f-49bb-925f-913d3c4c729a-root\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003143 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003109 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvcq\" (UniqueName: \"kubernetes.io/projected/0a393f07-726f-49bb-925f-913d3c4c729a-kube-api-access-ctvcq\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003143 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003147 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a393f07-726f-49bb-925f-913d3c4c729a-root\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003180 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-wtmp\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003198 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003216 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-accelerators-collector-config\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003236 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a393f07-726f-49bb-925f-913d3c4c729a-metrics-client-ca\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003248 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a393f07-726f-49bb-925f-913d3c4c729a-root\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003256 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a393f07-726f-49bb-925f-913d3c4c729a-sys\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003302 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a393f07-726f-49bb-925f-913d3c4c729a-sys\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003343 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-wtmp\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003357 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-textfile\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003404 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-tls\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.003974 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.003911 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-textfile\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.004062 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.004000 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-accelerators-collector-config\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.004062 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.004028 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a393f07-726f-49bb-925f-913d3c4c729a-metrics-client-ca\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.005887 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.005861 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.006007 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.005942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a393f07-726f-49bb-925f-913d3c4c729a-node-exporter-tls\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.025659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.025626 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvcq\" (UniqueName: \"kubernetes.io/projected/0a393f07-726f-49bb-925f-913d3c4c729a-kube-api-access-ctvcq\") pod \"node-exporter-g9nnw\" (UID: \"0a393f07-726f-49bb-925f-913d3c4c729a\") " pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.175178 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.175138 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g9nnw" Oct 30 00:34:45.218841 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:34:45.218808 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a393f07_726f_49bb_925f_913d3c4c729a.slice/crio-c2801c2b4cabe22479823c1c3922f38ad98f99e861a75c4c80e50253daa4fc76 WatchSource:0}: Error finding container c2801c2b4cabe22479823c1c3922f38ad98f99e861a75c4c80e50253daa4fc76: Status 404 returned error can't find the container with id c2801c2b4cabe22479823c1c3922f38ad98f99e861a75c4c80e50253daa4fc76 Oct 30 00:34:45.759217 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:45.759174 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g9nnw" event={"ID":"0a393f07-726f-49bb-925f-913d3c4c729a","Type":"ContainerStarted","Data":"c2801c2b4cabe22479823c1c3922f38ad98f99e861a75c4c80e50253daa4fc76"} Oct 30 00:34:46.764903 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:46.764816 4927 generic.go:356] "Generic (PLEG): container finished" podID="0a393f07-726f-49bb-925f-913d3c4c729a" containerID="283228501ceb39fbe8486906ac54a3ed25ddf9b22251d2493525caf76b866125" exitCode=0 Oct 30 00:34:46.764903 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:46.764872 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g9nnw" event={"ID":"0a393f07-726f-49bb-925f-913d3c4c729a","Type":"ContainerDied","Data":"283228501ceb39fbe8486906ac54a3ed25ddf9b22251d2493525caf76b866125"} Oct 30 00:34:46.960885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:46.960835 4927 patch_prober.go:29] interesting pod/image-registry-77dbcdb46-8kvk8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 00:34:46.961052 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:46.960913 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" podUID="afc047e1-508a-4f05-ae89-58db39eb5bdf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:47.771464 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:47.771429 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g9nnw" event={"ID":"0a393f07-726f-49bb-925f-913d3c4c729a","Type":"ContainerStarted","Data":"1b3130c8c75f443be81fe7ab6531f1033f5062b7ef7fea1c2a2e50f538599673"} Oct 30 00:34:47.771895 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:47.771484 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g9nnw" event={"ID":"0a393f07-726f-49bb-925f-913d3c4c729a","Type":"ContainerStarted","Data":"47da128e619f6f65a5d7f24a6842bb5f7de3d42a8f244668491fad3da6e14e96"} Oct 30 00:34:47.835701 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:47.835643 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g9nnw" podStartSLOduration=2.55404363 podStartE2EDuration="3.835628719s" totalImagesPullingTime="1.191370395s" totalInitContainerRuntime="90.214694ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:47.82368285 +0000 UTC m=+26.395609072" watchObservedRunningTime="2026-10-30 00:34:47.835628719 +0000 UTC m=+26.407554942" Oct 30 00:34:48.760245 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:48.760205 4927 patch_prober.go:29] interesting pod/image-registry-6b6dd9b44c-l6dl6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 00:34:48.760449 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:48.760266 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" podUID="89b792bc-b5bc-4153-86dc-65f1b7187a5a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:34:50.742593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:50.742550 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gbrwr" Oct 30 00:34:51.995981 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:51.995945 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wzp7p" Oct 30 00:34:56.960711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:56.960676 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:34:58.432856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:58.432820 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77dbcdb46-8kvk8"] Oct 30 00:34:58.758858 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:34:58.758786 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:35:01.421017 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.420950 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" podUID="89b792bc-b5bc-4153-86dc-65f1b7187a5a" containerName="registry" containerID="cri-o://363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8" gracePeriod=30 Oct 30 00:35:01.718780 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.718745 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:35:01.828107 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.828074 4927 generic.go:356] "Generic (PLEG): container finished" podID="89b792bc-b5bc-4153-86dc-65f1b7187a5a" containerID="363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8" exitCode=0 Oct 30 00:35:01.828301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.828118 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" event={"ID":"89b792bc-b5bc-4153-86dc-65f1b7187a5a","Type":"ContainerDied","Data":"363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8"} Oct 30 00:35:01.828301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.828134 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" Oct 30 00:35:01.828301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.828159 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6dd9b44c-l6dl6" event={"ID":"89b792bc-b5bc-4153-86dc-65f1b7187a5a","Type":"ContainerDied","Data":"659a6e1daa8b4e933270b45c78b0d1bfc28b293d12c9614733a5346173252068"} Oct 30 00:35:01.828301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.828177 4927 scope.go:122] "RemoveContainer" containerID="363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8" Oct 30 00:35:01.837854 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.837835 4927 scope.go:122] "RemoveContainer" containerID="363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8" Oct 30 00:35:01.838186 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:35:01.838163 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8\": container with ID starting with 363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8 not found: ID does not exist" containerID="363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8" Oct 30 00:35:01.838247 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.838199 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8"} err="failed to get container status \"363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8\": rpc error: code = NotFound desc = could not find container \"363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8\": container with ID starting with 363de9a3b5a5504c89e649e3fb8950a130426c61858c556ed96928b82d7c27c8 not found: ID does not exist" Oct 30 00:35:01.850871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.850839 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates\") pod \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " Oct 30 00:35:01.851018 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.850898 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets\") pod \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " Oct 30 00:35:01.851018 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.850929 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration\") pod \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " Oct 30 00:35:01.851018 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.850974 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted\") pod \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " Oct 30 00:35:01.851018 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.851008 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq\") pod \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " Oct 30 00:35:01.851200 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.851042 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca\") pod \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " Oct 30 00:35:01.851200 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.851090 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token\") pod \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " Oct 30 00:35:01.851200 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.851124 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") pod \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\" (UID: \"89b792bc-b5bc-4153-86dc-65f1b7187a5a\") " Oct 30 00:35:01.851405 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.851375 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates" pod "89b792bc-b5bc-4153-86dc-65f1b7187a5a" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:35:01.851978 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.851945 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca" pod "89b792bc-b5bc-4153-86dc-65f1b7187a5a" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:35:01.853856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.853823 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets" pod "89b792bc-b5bc-4153-86dc-65f1b7187a5a" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:35:01.853951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.853871 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls" pod "89b792bc-b5bc-4153-86dc-65f1b7187a5a" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:01.853951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.853915 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token" pod "89b792bc-b5bc-4153-86dc-65f1b7187a5a" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:01.854046 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.853969 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration" pod "89b792bc-b5bc-4153-86dc-65f1b7187a5a" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:35:01.854208 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.854186 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq" pod "89b792bc-b5bc-4153-86dc-65f1b7187a5a" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a"). InnerVolumeSpecName "kube-api-access-twbcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:01.861606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.861557 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted" pod "89b792bc-b5bc-4153-86dc-65f1b7187a5a" (UID: "89b792bc-b5bc-4153-86dc-65f1b7187a5a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:01.952469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.952387 4927 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-bound-sa-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:01.952469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.952419 4927 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:01.952469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.952428 4927 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-registry-certificates\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:01.952469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.952438 4927 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-installation-pull-secrets\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:01.952469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.952449 4927 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/89b792bc-b5bc-4153-86dc-65f1b7187a5a-image-registry-private-configuration\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:01.952469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.952458 4927 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/89b792bc-b5bc-4153-86dc-65f1b7187a5a-ca-trust-extracted\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:01.952469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.952467 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twbcq\" (UniqueName: \"kubernetes.io/projected/89b792bc-b5bc-4153-86dc-65f1b7187a5a-kube-api-access-twbcq\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:01.952469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:01.952476 4927 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89b792bc-b5bc-4153-86dc-65f1b7187a5a-trusted-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:02.172709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:02.172672 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b6dd9b44c-l6dl6"] Oct 30 00:35:02.184115 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:02.184087 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b6dd9b44c-l6dl6"] Oct 30 00:35:03.568954 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:03.568873 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b792bc-b5bc-4153-86dc-65f1b7187a5a" path="/var/lib/kubelet/pods/89b792bc-b5bc-4153-86dc-65f1b7187a5a/volumes" Oct 30 00:35:04.441078 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:35:04.441043 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:35:23.452903 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.452839 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" podUID="afc047e1-508a-4f05-ae89-58db39eb5bdf" containerName="registry" containerID="cri-o://7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb" gracePeriod=30 Oct 30 00:35:23.695364 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.695337 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:35:23.835855 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.835770 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration\") pod \"afc047e1-508a-4f05-ae89-58db39eb5bdf\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " Oct 30 00:35:23.835855 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.835825 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") pod \"afc047e1-508a-4f05-ae89-58db39eb5bdf\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " Oct 30 00:35:23.835855 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.835853 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca\") pod \"afc047e1-508a-4f05-ae89-58db39eb5bdf\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " Oct 30 00:35:23.836153 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.835875 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates\") pod \"afc047e1-508a-4f05-ae89-58db39eb5bdf\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " Oct 30 00:35:23.836153 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.835929 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted\") pod \"afc047e1-508a-4f05-ae89-58db39eb5bdf\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " Oct 30 00:35:23.836153 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.835954 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token\") pod \"afc047e1-508a-4f05-ae89-58db39eb5bdf\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " Oct 30 00:35:23.836153 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.835990 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets\") pod \"afc047e1-508a-4f05-ae89-58db39eb5bdf\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " Oct 30 00:35:23.836153 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.836028 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s\") pod \"afc047e1-508a-4f05-ae89-58db39eb5bdf\" (UID: \"afc047e1-508a-4f05-ae89-58db39eb5bdf\") " Oct 30 00:35:23.836428 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.836398 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates" pod "afc047e1-508a-4f05-ae89-58db39eb5bdf" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:35:23.836484 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.836405 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca" pod "afc047e1-508a-4f05-ae89-58db39eb5bdf" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:35:23.838987 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.838955 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration" pod "afc047e1-508a-4f05-ae89-58db39eb5bdf" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:35:23.839124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.838980 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls" pod "afc047e1-508a-4f05-ae89-58db39eb5bdf" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:23.839124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.838997 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s" pod "afc047e1-508a-4f05-ae89-58db39eb5bdf" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf"). InnerVolumeSpecName "kube-api-access-djt7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:23.839225 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.839110 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets" pod "afc047e1-508a-4f05-ae89-58db39eb5bdf" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:35:23.839225 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.839183 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token" pod "afc047e1-508a-4f05-ae89-58db39eb5bdf" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:35:23.845527 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.845488 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted" pod "afc047e1-508a-4f05-ae89-58db39eb5bdf" (UID: "afc047e1-508a-4f05-ae89-58db39eb5bdf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:35:23.919029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.918993 4927 generic.go:356] "Generic (PLEG): container finished" podID="afc047e1-508a-4f05-ae89-58db39eb5bdf" containerID="7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb" exitCode=0 Oct 30 00:35:23.919230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.919048 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" Oct 30 00:35:23.919230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.919083 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" event={"ID":"afc047e1-508a-4f05-ae89-58db39eb5bdf","Type":"ContainerDied","Data":"7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb"} Oct 30 00:35:23.919230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.919144 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77dbcdb46-8kvk8" event={"ID":"afc047e1-508a-4f05-ae89-58db39eb5bdf","Type":"ContainerDied","Data":"7cedda7c0a487d02309cf2191e1507ec9eb2092817dba562a56bc0f3e85a9802"} Oct 30 00:35:23.919230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.919169 4927 scope.go:122] "RemoveContainer" containerID="7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb" Oct 30 00:35:23.927968 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.927945 4927 scope.go:122] "RemoveContainer" containerID="7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb" Oct 30 00:35:23.928234 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:35:23.928216 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb\": container with ID starting with 7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb not found: ID does not exist" containerID="7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb" Oct 30 00:35:23.928278 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.928243 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb"} err="failed to get container status \"7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb\": rpc error: code = NotFound desc = could not find container \"7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb\": container with ID starting with 7635b0ada8f1818778ea64687c7800118b7d8788b8bab7637b2580994eca16bb not found: ID does not exist" Oct 30 00:35:23.937336 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.937314 4927 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:23.937336 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.937336 4927 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-trusted-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:23.937489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.937346 4927 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afc047e1-508a-4f05-ae89-58db39eb5bdf-registry-certificates\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:23.937489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.937356 4927 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afc047e1-508a-4f05-ae89-58db39eb5bdf-ca-trust-extracted\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:23.937489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.937367 4927 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-bound-sa-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:23.937489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.937376 4927 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-installation-pull-secrets\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:23.937489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.937384 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djt7s\" (UniqueName: \"kubernetes.io/projected/afc047e1-508a-4f05-ae89-58db39eb5bdf-kube-api-access-djt7s\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:23.937489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.937394 4927 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afc047e1-508a-4f05-ae89-58db39eb5bdf-image-registry-private-configuration\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:35:23.966852 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.966808 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77dbcdb46-8kvk8"] Oct 30 00:35:23.970620 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:23.970591 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77dbcdb46-8kvk8"] Oct 30 00:35:25.568468 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:25.568431 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc047e1-508a-4f05-ae89-58db39eb5bdf" path="/var/lib/kubelet/pods/afc047e1-508a-4f05-ae89-58db39eb5bdf/volumes" Oct 30 00:35:31.953695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:31.953657 4927 generic.go:356] "Generic (PLEG): container finished" podID="104d94b9-331d-4121-9f9f-61c191d85bba" containerID="592d79ffa2135db3907ea3ff50f370305530ec117cdb1719589ba7c5bf937e35" exitCode=0 Oct 30 00:35:31.954193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:31.953733 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tvh9g" event={"ID":"104d94b9-331d-4121-9f9f-61c191d85bba","Type":"ContainerDied","Data":"592d79ffa2135db3907ea3ff50f370305530ec117cdb1719589ba7c5bf937e35"} Oct 30 00:35:31.954193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:31.954146 4927 scope.go:122] "RemoveContainer" containerID="592d79ffa2135db3907ea3ff50f370305530ec117cdb1719589ba7c5bf937e35" Oct 30 00:35:32.959890 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:32.959854 4927 generic.go:356] "Generic (PLEG): container finished" podID="1d35d272-2cbb-4667-beb8-252aec93d74c" containerID="efe0f54ce11ba31f3ed854c95f082fa1b61c9d0ecc7969084fa4e9876da6cbe7" exitCode=0 Oct 30 00:35:32.960313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:32.959921 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" event={"ID":"1d35d272-2cbb-4667-beb8-252aec93d74c","Type":"ContainerDied","Data":"efe0f54ce11ba31f3ed854c95f082fa1b61c9d0ecc7969084fa4e9876da6cbe7"} Oct 30 00:35:32.960379 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:32.960365 4927 scope.go:122] "RemoveContainer" containerID="efe0f54ce11ba31f3ed854c95f082fa1b61c9d0ecc7969084fa4e9876da6cbe7" Oct 30 00:35:32.962097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:32.962075 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tvh9g" event={"ID":"104d94b9-331d-4121-9f9f-61c191d85bba","Type":"ContainerStarted","Data":"2e332478c25f3f9ff8e096ec6ec6461eff1a989851fe0e79d1a4428ccaf0d413"} Oct 30 00:35:33.967971 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:33.967935 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sc9sj" event={"ID":"1d35d272-2cbb-4667-beb8-252aec93d74c","Type":"ContainerStarted","Data":"9898ef3a56f2b27e1b77510e29603f22460a8245a908e1cdcb64e96622631904"} Oct 30 00:35:34.441259 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:35:34.441218 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:35:57.058973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:57.058882 4927 generic.go:356] "Generic (PLEG): container finished" podID="1a5c79ec-41c6-4613-a010-af274b5f21ac" containerID="8e518e24bee9cb7c0a1f60a131e0a68122fa3bed923d2b72d186a7ed67481463" exitCode=0 Oct 30 00:35:57.058973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:57.058951 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" event={"ID":"1a5c79ec-41c6-4613-a010-af274b5f21ac","Type":"ContainerDied","Data":"8e518e24bee9cb7c0a1f60a131e0a68122fa3bed923d2b72d186a7ed67481463"} Oct 30 00:35:57.059430 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:57.059268 4927 scope.go:122] "RemoveContainer" containerID="8e518e24bee9cb7c0a1f60a131e0a68122fa3bed923d2b72d186a7ed67481463" Oct 30 00:35:58.065364 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:35:58.065327 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-m5w79" event={"ID":"1a5c79ec-41c6-4613-a010-af274b5f21ac","Type":"ContainerStarted","Data":"a37ffb2f720f7f7199c39434b6c7588934c51767bdd7401631b54e266ec5700e"} Oct 30 00:36:04.441381 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:36:04.441338 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:36:34.442507 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:36:34.442420 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:37:04.441302 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:37:04.441266 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:37:23.837927 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.837876 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb"] Oct 30 00:37:23.838402 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.838303 4927 cpu_manager.go:414] "containerMap: removing container" podUID="afc047e1-508a-4f05-ae89-58db39eb5bdf" containerName="registry" Oct 30 00:37:23.838402 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.838319 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc047e1-508a-4f05-ae89-58db39eb5bdf" containerName="registry" Oct 30 00:37:23.838402 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.838334 4927 cpu_manager.go:414] "containerMap: removing container" podUID="89b792bc-b5bc-4153-86dc-65f1b7187a5a" containerName="registry" Oct 30 00:37:23.838402 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.838339 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b792bc-b5bc-4153-86dc-65f1b7187a5a" containerName="registry" Oct 30 00:37:23.838402 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.838398 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc047e1-508a-4f05-ae89-58db39eb5bdf" containerName="registry" Oct 30 00:37:23.838557 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.838407 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b792bc-b5bc-4153-86dc-65f1b7187a5a" containerName="registry" Oct 30 00:37:23.841559 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.841535 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:23.869491 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.869453 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 00:37:23.869491 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.869467 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 00:37:23.869774 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.869525 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-mmczf\"" Oct 30 00:37:23.870620 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.870597 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb"] Oct 30 00:37:23.995668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.995627 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:23.995846 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.995698 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlc9x\" (UniqueName: \"kubernetes.io/projected/f97d3759-e747-43cd-91fa-474f86c0e874-kube-api-access-zlc9x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:23.995846 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:23.995810 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:24.096827 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:24.096716 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:24.096827 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:24.096794 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlc9x\" (UniqueName: \"kubernetes.io/projected/f97d3759-e747-43cd-91fa-474f86c0e874-kube-api-access-zlc9x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:24.097082 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:24.096842 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:24.097146 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:24.097129 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:24.097211 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:24.097195 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:24.119954 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:24.119917 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlc9x\" (UniqueName: \"kubernetes.io/projected/f97d3759-e747-43cd-91fa-474f86c0e874-kube-api-access-zlc9x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:24.161882 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:24.161844 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:24.459029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:24.458992 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb"] Oct 30 00:37:24.462909 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:37:24.462880 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97d3759_e747_43cd_91fa_474f86c0e874.slice/crio-c73afd8cc19ffeabebb440cbbd3c7b820d52580351ffe3acd9f10ec4fae81215 WatchSource:0}: Error finding container c73afd8cc19ffeabebb440cbbd3c7b820d52580351ffe3acd9f10ec4fae81215: Status 404 returned error can't find the container with id c73afd8cc19ffeabebb440cbbd3c7b820d52580351ffe3acd9f10ec4fae81215 Oct 30 00:37:25.391361 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:25.391312 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" event={"ID":"f97d3759-e747-43cd-91fa-474f86c0e874","Type":"ContainerStarted","Data":"c73afd8cc19ffeabebb440cbbd3c7b820d52580351ffe3acd9f10ec4fae81215"} Oct 30 00:37:30.414380 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:30.414336 4927 generic.go:356] "Generic (PLEG): container finished" podID="f97d3759-e747-43cd-91fa-474f86c0e874" containerID="98e72bedf9621e2cf3693b860a78f9d5c0836ad2bc1758833798939b42f8b639" exitCode=0 Oct 30 00:37:30.414910 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:30.414423 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" event={"ID":"f97d3759-e747-43cd-91fa-474f86c0e874","Type":"ContainerDied","Data":"98e72bedf9621e2cf3693b860a78f9d5c0836ad2bc1758833798939b42f8b639"} Oct 30 00:37:32.425504 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:32.425469 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" event={"ID":"f97d3759-e747-43cd-91fa-474f86c0e874","Type":"ContainerStarted","Data":"e484a7ad186f129d308740118120f3661ac94fdf626a2e34ce4888c1633c9bae"} Oct 30 00:37:33.431330 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:33.431292 4927 generic.go:356] "Generic (PLEG): container finished" podID="f97d3759-e747-43cd-91fa-474f86c0e874" containerID="e484a7ad186f129d308740118120f3661ac94fdf626a2e34ce4888c1633c9bae" exitCode=0 Oct 30 00:37:33.431794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:33.431362 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" event={"ID":"f97d3759-e747-43cd-91fa-474f86c0e874","Type":"ContainerDied","Data":"e484a7ad186f129d308740118120f3661ac94fdf626a2e34ce4888c1633c9bae"} Oct 30 00:37:34.447390 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:37:34.447322 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:37:40.465732 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:40.465697 4927 generic.go:356] "Generic (PLEG): container finished" podID="f97d3759-e747-43cd-91fa-474f86c0e874" containerID="9289c677a3299c1ae420313af964e16b41ac1fbf9686161182e2fe3ab9af05bf" exitCode=0 Oct 30 00:37:40.466134 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:40.465798 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" event={"ID":"f97d3759-e747-43cd-91fa-474f86c0e874","Type":"ContainerDied","Data":"9289c677a3299c1ae420313af964e16b41ac1fbf9686161182e2fe3ab9af05bf"} Oct 30 00:37:41.596581 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.596550 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:41.647308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.647275 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97d3759-e747-43cd-91fa-474f86c0e874-kube-api-access-zlc9x\" (UniqueName: \"kubernetes.io/projected/f97d3759-e747-43cd-91fa-474f86c0e874-kube-api-access-zlc9x\") pod \"f97d3759-e747-43cd-91fa-474f86c0e874\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " Oct 30 00:37:41.649581 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.649543 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97d3759-e747-43cd-91fa-474f86c0e874-kube-api-access-zlc9x" pod "f97d3759-e747-43cd-91fa-474f86c0e874" (UID: "f97d3759-e747-43cd-91fa-474f86c0e874"). InnerVolumeSpecName "kube-api-access-zlc9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:41.747860 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.747775 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-util\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-util\") pod \"f97d3759-e747-43cd-91fa-474f86c0e874\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " Oct 30 00:37:41.748024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.747879 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-bundle\") pod \"f97d3759-e747-43cd-91fa-474f86c0e874\" (UID: \"f97d3759-e747-43cd-91fa-474f86c0e874\") " Oct 30 00:37:41.748024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.747975 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlc9x\" (UniqueName: \"kubernetes.io/projected/f97d3759-e747-43cd-91fa-474f86c0e874-kube-api-access-zlc9x\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:37:41.748459 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.748434 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-bundle" pod "f97d3759-e747-43cd-91fa-474f86c0e874" (UID: "f97d3759-e747-43cd-91fa-474f86c0e874"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:41.751749 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.751723 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-util" pod "f97d3759-e747-43cd-91fa-474f86c0e874" (UID: "f97d3759-e747-43cd-91fa-474f86c0e874"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:41.848812 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.848760 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:37:41.848812 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:41.848806 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f97d3759-e747-43cd-91fa-474f86c0e874-util\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:37:42.475548 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:42.475519 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" Oct 30 00:37:42.475756 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:42.475512 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crrxtb" event={"ID":"f97d3759-e747-43cd-91fa-474f86c0e874","Type":"ContainerDied","Data":"c73afd8cc19ffeabebb440cbbd3c7b820d52580351ffe3acd9f10ec4fae81215"} Oct 30 00:37:42.475756 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:42.475634 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c73afd8cc19ffeabebb440cbbd3c7b820d52580351ffe3acd9f10ec4fae81215" Oct 30 00:37:46.879195 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.879163 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6"] Oct 30 00:37:46.879757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.879457 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f97d3759-e747-43cd-91fa-474f86c0e874" containerName="pull" Oct 30 00:37:46.879757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.879469 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97d3759-e747-43cd-91fa-474f86c0e874" containerName="pull" Oct 30 00:37:46.879757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.879490 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f97d3759-e747-43cd-91fa-474f86c0e874" containerName="extract" Oct 30 00:37:46.879757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.879495 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97d3759-e747-43cd-91fa-474f86c0e874" containerName="extract" Oct 30 00:37:46.879757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.879506 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f97d3759-e747-43cd-91fa-474f86c0e874" containerName="util" Oct 30 00:37:46.879757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.879511 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97d3759-e747-43cd-91fa-474f86c0e874" containerName="util" Oct 30 00:37:46.879757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.879561 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97d3759-e747-43cd-91fa-474f86c0e874" containerName="extract" Oct 30 00:37:46.917457 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.917419 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6"] Oct 30 00:37:46.917650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.917547 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" Oct 30 00:37:46.931527 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.931500 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:37:46.931692 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.931502 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 00:37:46.942775 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.942750 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rddpk\"" Oct 30 00:37:46.982924 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.982885 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da1df003-ed01-431e-953c-4967d8ebd67d-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cmvt6\" (UID: \"da1df003-ed01-431e-953c-4967d8ebd67d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" Oct 30 00:37:46.982924 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:46.982929 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6gb4\" (UniqueName: \"kubernetes.io/projected/da1df003-ed01-431e-953c-4967d8ebd67d-kube-api-access-x6gb4\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cmvt6\" (UID: \"da1df003-ed01-431e-953c-4967d8ebd67d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" Oct 30 00:37:47.083505 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:47.083464 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da1df003-ed01-431e-953c-4967d8ebd67d-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cmvt6\" (UID: \"da1df003-ed01-431e-953c-4967d8ebd67d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" Oct 30 00:37:47.083738 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:47.083522 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6gb4\" (UniqueName: \"kubernetes.io/projected/da1df003-ed01-431e-953c-4967d8ebd67d-kube-api-access-x6gb4\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cmvt6\" (UID: \"da1df003-ed01-431e-953c-4967d8ebd67d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" Oct 30 00:37:47.083917 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:47.083892 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da1df003-ed01-431e-953c-4967d8ebd67d-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cmvt6\" (UID: \"da1df003-ed01-431e-953c-4967d8ebd67d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" Oct 30 00:37:47.106597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:47.106538 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6gb4\" (UniqueName: \"kubernetes.io/projected/da1df003-ed01-431e-953c-4967d8ebd67d-kube-api-access-x6gb4\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cmvt6\" (UID: \"da1df003-ed01-431e-953c-4967d8ebd67d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" Oct 30 00:37:47.239517 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:47.239425 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" Oct 30 00:37:47.590339 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:47.590261 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6"] Oct 30 00:37:47.593949 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:37:47.593912 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1df003_ed01_431e_953c_4967d8ebd67d.slice/crio-d3ec9417931a349e9715899032032ad6d960903b62c43a65a26488de8577f407 WatchSource:0}: Error finding container d3ec9417931a349e9715899032032ad6d960903b62c43a65a26488de8577f407: Status 404 returned error can't find the container with id d3ec9417931a349e9715899032032ad6d960903b62c43a65a26488de8577f407 Oct 30 00:37:48.501991 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:48.501947 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" event={"ID":"da1df003-ed01-431e-953c-4967d8ebd67d","Type":"ContainerStarted","Data":"d3ec9417931a349e9715899032032ad6d960903b62c43a65a26488de8577f407"} Oct 30 00:37:50.511083 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:50.511041 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" event={"ID":"da1df003-ed01-431e-953c-4967d8ebd67d","Type":"ContainerStarted","Data":"02851ad8078059660902bfa78f12df50e63cb78cdfe8ff6bf784664636dc8fd5"} Oct 30 00:37:50.576020 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:50.575966 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cmvt6" podStartSLOduration=2.423785011 podStartE2EDuration="4.575950091s" totalImagesPullingTime="2.15216508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:50.564392193 +0000 UTC m=+209.136318414" watchObservedRunningTime="2026-10-30 00:37:50.575950091 +0000 UTC m=+209.147876309" Oct 30 00:37:51.423035 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.422996 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c"] Oct 30 00:37:51.446374 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.446332 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.448102 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.448061 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c"] Oct 30 00:37:51.471684 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.471650 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 00:37:51.471684 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.471657 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 00:37:51.471905 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.471694 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-gsm9h\"" Oct 30 00:37:51.521009 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.520976 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.521425 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.521020 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.521425 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.521059 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rbq\" (UniqueName: \"kubernetes.io/projected/96c2dba2-9759-4161-abef-063af86f34de-kube-api-access-r7rbq\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.622395 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.622351 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.622632 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.622438 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rbq\" (UniqueName: \"kubernetes.io/projected/96c2dba2-9759-4161-abef-063af86f34de-kube-api-access-r7rbq\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.622632 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.622488 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.622802 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.622777 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.622889 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.622871 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.654656 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.654617 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rbq\" (UniqueName: \"kubernetes.io/projected/96c2dba2-9759-4161-abef-063af86f34de-kube-api-access-r7rbq\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:51.798963 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:51.798871 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:52.110945 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:52.110855 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c"] Oct 30 00:37:52.114300 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:37:52.114268 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c2dba2_9759_4161_abef_063af86f34de.slice/crio-c9ddb9a09133f88bc312998e106870b4c2f7998f31dae78414dde72cd3d523cc WatchSource:0}: Error finding container c9ddb9a09133f88bc312998e106870b4c2f7998f31dae78414dde72cd3d523cc: Status 404 returned error can't find the container with id c9ddb9a09133f88bc312998e106870b4c2f7998f31dae78414dde72cd3d523cc Oct 30 00:37:52.520927 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:52.520891 4927 generic.go:356] "Generic (PLEG): container finished" podID="96c2dba2-9759-4161-abef-063af86f34de" containerID="9ee05a829412c53d26dc0e16d0473dfd85034660661b6631e4cd3e22b3004124" exitCode=0 Oct 30 00:37:52.521129 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:52.520964 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" event={"ID":"96c2dba2-9759-4161-abef-063af86f34de","Type":"ContainerDied","Data":"9ee05a829412c53d26dc0e16d0473dfd85034660661b6631e4cd3e22b3004124"} Oct 30 00:37:52.521129 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:52.521031 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" event={"ID":"96c2dba2-9759-4161-abef-063af86f34de","Type":"ContainerStarted","Data":"c9ddb9a09133f88bc312998e106870b4c2f7998f31dae78414dde72cd3d523cc"} Oct 30 00:37:55.536957 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:55.536923 4927 generic.go:356] "Generic (PLEG): container finished" podID="96c2dba2-9759-4161-abef-063af86f34de" containerID="24052e9c81bbc02a7b76161046cff489091d08d02880ca16f95e732b7f7f1700" exitCode=0 Oct 30 00:37:55.537368 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:55.536985 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" event={"ID":"96c2dba2-9759-4161-abef-063af86f34de","Type":"ContainerDied","Data":"24052e9c81bbc02a7b76161046cff489091d08d02880ca16f95e732b7f7f1700"} Oct 30 00:37:56.543237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:56.543199 4927 generic.go:356] "Generic (PLEG): container finished" podID="96c2dba2-9759-4161-abef-063af86f34de" containerID="9cd811d9f690a603a3b17fe899e2f766cace6dc29e25f46e875b0d6c874ad643" exitCode=0 Oct 30 00:37:56.543695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:56.543250 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" event={"ID":"96c2dba2-9759-4161-abef-063af86f34de","Type":"ContainerDied","Data":"9cd811d9f690a603a3b17fe899e2f766cace6dc29e25f46e875b0d6c874ad643"} Oct 30 00:37:57.562911 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.562876 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw"] Oct 30 00:37:57.567116 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.567086 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" Oct 30 00:37:57.583875 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.583847 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 00:37:57.592412 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.592383 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 00:37:57.592412 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.592384 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-c6dvw\"" Oct 30 00:37:57.593543 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.593525 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw"] Oct 30 00:37:57.669213 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.669178 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f73b373f-6edc-49a8-92cb-5314f46ed972-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-c8pvw\" (UID: \"f73b373f-6edc-49a8-92cb-5314f46ed972\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" Oct 30 00:37:57.669366 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.669223 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm79c\" (UniqueName: \"kubernetes.io/projected/f73b373f-6edc-49a8-92cb-5314f46ed972-kube-api-access-mm79c\") pod \"cert-manager-cainjector-5c57bbd4fc-c8pvw\" (UID: \"f73b373f-6edc-49a8-92cb-5314f46ed972\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" Oct 30 00:37:57.674826 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.674803 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:57.769845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.769806 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c2dba2-9759-4161-abef-063af86f34de-kube-api-access-r7rbq\" (UniqueName: \"kubernetes.io/projected/96c2dba2-9759-4161-abef-063af86f34de-kube-api-access-r7rbq\") pod \"96c2dba2-9759-4161-abef-063af86f34de\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " Oct 30 00:37:57.770054 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.769865 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-util\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-util\") pod \"96c2dba2-9759-4161-abef-063af86f34de\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " Oct 30 00:37:57.770054 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.769899 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-bundle\") pod \"96c2dba2-9759-4161-abef-063af86f34de\" (UID: \"96c2dba2-9759-4161-abef-063af86f34de\") " Oct 30 00:37:57.770054 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.770040 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f73b373f-6edc-49a8-92cb-5314f46ed972-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-c8pvw\" (UID: \"f73b373f-6edc-49a8-92cb-5314f46ed972\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" Oct 30 00:37:57.770221 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.770072 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm79c\" (UniqueName: \"kubernetes.io/projected/f73b373f-6edc-49a8-92cb-5314f46ed972-kube-api-access-mm79c\") pod \"cert-manager-cainjector-5c57bbd4fc-c8pvw\" (UID: \"f73b373f-6edc-49a8-92cb-5314f46ed972\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" Oct 30 00:37:57.770361 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.770337 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-bundle" pod "96c2dba2-9759-4161-abef-063af86f34de" (UID: "96c2dba2-9759-4161-abef-063af86f34de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:57.772184 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.772155 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c2dba2-9759-4161-abef-063af86f34de-kube-api-access-r7rbq" pod "96c2dba2-9759-4161-abef-063af86f34de" (UID: "96c2dba2-9759-4161-abef-063af86f34de"). InnerVolumeSpecName "kube-api-access-r7rbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:37:57.775186 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.775160 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-util" pod "96c2dba2-9759-4161-abef-063af86f34de" (UID: "96c2dba2-9759-4161-abef-063af86f34de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:37:57.794121 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.794093 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f73b373f-6edc-49a8-92cb-5314f46ed972-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-c8pvw\" (UID: \"f73b373f-6edc-49a8-92cb-5314f46ed972\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" Oct 30 00:37:57.794344 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.794318 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm79c\" (UniqueName: \"kubernetes.io/projected/f73b373f-6edc-49a8-92cb-5314f46ed972-kube-api-access-mm79c\") pod \"cert-manager-cainjector-5c57bbd4fc-c8pvw\" (UID: \"f73b373f-6edc-49a8-92cb-5314f46ed972\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" Oct 30 00:37:57.871560 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.871474 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rbq\" (UniqueName: \"kubernetes.io/projected/96c2dba2-9759-4161-abef-063af86f34de-kube-api-access-r7rbq\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:37:57.871560 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.871513 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-util\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:37:57.871560 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.871523 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96c2dba2-9759-4161-abef-063af86f34de-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:37:57.889461 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:57.889422 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" Oct 30 00:37:58.187754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:58.187718 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw"] Oct 30 00:37:58.190888 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:37:58.190853 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf73b373f_6edc_49a8_92cb_5314f46ed972.slice/crio-25cf869c10e6983c79570b675edf9794cfb93f4576e4cd4c9e042ac096e0b7da WatchSource:0}: Error finding container 25cf869c10e6983c79570b675edf9794cfb93f4576e4cd4c9e042ac096e0b7da: Status 404 returned error can't find the container with id 25cf869c10e6983c79570b675edf9794cfb93f4576e4cd4c9e042ac096e0b7da Oct 30 00:37:58.553086 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:58.552999 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" event={"ID":"96c2dba2-9759-4161-abef-063af86f34de","Type":"ContainerDied","Data":"c9ddb9a09133f88bc312998e106870b4c2f7998f31dae78414dde72cd3d523cc"} Oct 30 00:37:58.553271 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:58.553013 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85hjb8c" Oct 30 00:37:58.553271 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:58.553096 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ddb9a09133f88bc312998e106870b4c2f7998f31dae78414dde72cd3d523cc" Oct 30 00:37:58.554351 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:37:58.554317 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" event={"ID":"f73b373f-6edc-49a8-92cb-5314f46ed972","Type":"ContainerStarted","Data":"25cf869c10e6983c79570b675edf9794cfb93f4576e4cd4c9e042ac096e0b7da"} Oct 30 00:38:01.569660 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:01.569620 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" event={"ID":"f73b373f-6edc-49a8-92cb-5314f46ed972","Type":"ContainerStarted","Data":"951cb4b9425abe21ad7a54ca10141168651fe00bdff0cc8fa33f76cb4dbbbead"} Oct 30 00:38:01.607197 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:01.607133 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-c8pvw" podStartSLOduration=1.995091212 podStartE2EDuration="4.607119637s" totalImagesPullingTime="2.612028425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:01.606936099 +0000 UTC m=+220.178862323" watchObservedRunningTime="2026-10-30 00:38:01.607119637 +0000 UTC m=+220.179045858" Oct 30 00:38:04.441357 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:38:04.441269 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:38:13.134051 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.134013 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx"] Oct 30 00:38:13.134451 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.134330 4927 cpu_manager.go:414] "containerMap: removing container" podUID="96c2dba2-9759-4161-abef-063af86f34de" containerName="extract" Oct 30 00:38:13.134451 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.134341 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c2dba2-9759-4161-abef-063af86f34de" containerName="extract" Oct 30 00:38:13.134451 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.134359 4927 cpu_manager.go:414] "containerMap: removing container" podUID="96c2dba2-9759-4161-abef-063af86f34de" containerName="pull" Oct 30 00:38:13.134451 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.134364 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c2dba2-9759-4161-abef-063af86f34de" containerName="pull" Oct 30 00:38:13.134451 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.134377 4927 cpu_manager.go:414] "containerMap: removing container" podUID="96c2dba2-9759-4161-abef-063af86f34de" containerName="util" Oct 30 00:38:13.134451 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.134382 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c2dba2-9759-4161-abef-063af86f34de" containerName="util" Oct 30 00:38:13.134451 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.134434 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c2dba2-9759-4161-abef-063af86f34de" containerName="extract" Oct 30 00:38:13.137644 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.137623 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.162518 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.162487 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 00:38:13.162755 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.162561 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 00:38:13.162755 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.162694 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-cdr8n\"" Oct 30 00:38:13.164082 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.164046 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx"] Oct 30 00:38:13.194711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.194665 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwscl\" (UniqueName: \"kubernetes.io/projected/ce927abf-ce2b-4823-a37a-3360acd476a1-kube-api-access-hwscl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.194711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.194717 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.194939 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.194798 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.296230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.296194 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.296382 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.296293 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwscl\" (UniqueName: \"kubernetes.io/projected/ce927abf-ce2b-4823-a37a-3360acd476a1-kube-api-access-hwscl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.296382 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.296323 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.296752 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.296726 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.296799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.296754 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.319472 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.319440 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwscl\" (UniqueName: \"kubernetes.io/projected/ce927abf-ce2b-4823-a37a-3360acd476a1-kube-api-access-hwscl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.453635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.453536 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:13.753429 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:13.753344 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx"] Oct 30 00:38:13.756336 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:38:13.756295 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce927abf_ce2b_4823_a37a_3360acd476a1.slice/crio-547307538d1657277beef44df4d8dc9d60428f9b3a712e7f072f14389e0a3b99 WatchSource:0}: Error finding container 547307538d1657277beef44df4d8dc9d60428f9b3a712e7f072f14389e0a3b99: Status 404 returned error can't find the container with id 547307538d1657277beef44df4d8dc9d60428f9b3a712e7f072f14389e0a3b99 Oct 30 00:38:14.624048 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:14.624013 4927 generic.go:356] "Generic (PLEG): container finished" podID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerID="756df382c550be94cba0b709a4e6cf03c4c3a64eabf628cbd30725925e71b357" exitCode=0 Oct 30 00:38:14.624471 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:14.624114 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" event={"ID":"ce927abf-ce2b-4823-a37a-3360acd476a1","Type":"ContainerDied","Data":"756df382c550be94cba0b709a4e6cf03c4c3a64eabf628cbd30725925e71b357"} Oct 30 00:38:14.624471 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:14.624184 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" event={"ID":"ce927abf-ce2b-4823-a37a-3360acd476a1","Type":"ContainerStarted","Data":"547307538d1657277beef44df4d8dc9d60428f9b3a712e7f072f14389e0a3b99"} Oct 30 00:38:15.630461 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:15.630423 4927 generic.go:356] "Generic (PLEG): container finished" podID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerID="17de508035c16e425eab8278b80b84c43be9a494c82a668b986a6e784aa79054" exitCode=0 Oct 30 00:38:15.630888 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:15.630506 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" event={"ID":"ce927abf-ce2b-4823-a37a-3360acd476a1","Type":"ContainerDied","Data":"17de508035c16e425eab8278b80b84c43be9a494c82a668b986a6e784aa79054"} Oct 30 00:38:16.636729 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:16.636691 4927 generic.go:356] "Generic (PLEG): container finished" podID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerID="948b1c456505e6c37c12985933e697c2a35f0979053c43621cd395b6ec32609a" exitCode=0 Oct 30 00:38:16.637134 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:16.636780 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" event={"ID":"ce927abf-ce2b-4823-a37a-3360acd476a1","Type":"ContainerDied","Data":"948b1c456505e6c37c12985933e697c2a35f0979053c43621cd395b6ec32609a"} Oct 30 00:38:17.785287 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.785258 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:17.834290 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.834259 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-util\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-util\") pod \"ce927abf-ce2b-4823-a37a-3360acd476a1\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " Oct 30 00:38:17.834460 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.834340 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-bundle\") pod \"ce927abf-ce2b-4823-a37a-3360acd476a1\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " Oct 30 00:38:17.834460 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.834398 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce927abf-ce2b-4823-a37a-3360acd476a1-kube-api-access-hwscl\" (UniqueName: \"kubernetes.io/projected/ce927abf-ce2b-4823-a37a-3360acd476a1-kube-api-access-hwscl\") pod \"ce927abf-ce2b-4823-a37a-3360acd476a1\" (UID: \"ce927abf-ce2b-4823-a37a-3360acd476a1\") " Oct 30 00:38:17.834975 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.834942 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-bundle" pod "ce927abf-ce2b-4823-a37a-3360acd476a1" (UID: "ce927abf-ce2b-4823-a37a-3360acd476a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:17.836541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.836509 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce927abf-ce2b-4823-a37a-3360acd476a1-kube-api-access-hwscl" pod "ce927abf-ce2b-4823-a37a-3360acd476a1" (UID: "ce927abf-ce2b-4823-a37a-3360acd476a1"). InnerVolumeSpecName "kube-api-access-hwscl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:17.839500 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.839476 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-util" pod "ce927abf-ce2b-4823-a37a-3360acd476a1" (UID: "ce927abf-ce2b-4823-a37a-3360acd476a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:38:17.935707 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.935620 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-util\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:38:17.935707 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.935652 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce927abf-ce2b-4823-a37a-3360acd476a1-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:38:17.935707 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:17.935661 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwscl\" (UniqueName: \"kubernetes.io/projected/ce927abf-ce2b-4823-a37a-3360acd476a1-kube-api-access-hwscl\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:38:18.646827 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:18.646795 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" Oct 30 00:38:18.647020 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:18.646792 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7185gzx" event={"ID":"ce927abf-ce2b-4823-a37a-3360acd476a1","Type":"ContainerDied","Data":"547307538d1657277beef44df4d8dc9d60428f9b3a712e7f072f14389e0a3b99"} Oct 30 00:38:18.647020 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:18.646921 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="547307538d1657277beef44df4d8dc9d60428f9b3a712e7f072f14389e0a3b99" Oct 30 00:38:28.282072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.282033 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm"] Oct 30 00:38:28.282481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.282347 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerName="pull" Oct 30 00:38:28.282481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.282357 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerName="pull" Oct 30 00:38:28.282481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.282373 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerName="extract" Oct 30 00:38:28.282481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.282378 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerName="extract" Oct 30 00:38:28.282481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.282398 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerName="util" Oct 30 00:38:28.282481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.282403 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerName="util" Oct 30 00:38:28.282481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.282453 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce927abf-ce2b-4823-a37a-3360acd476a1" containerName="extract" Oct 30 00:38:28.290240 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.290215 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.318948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.318900 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 00:38:28.318948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.318912 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 00:38:28.318948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.318947 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 00:38:28.319235 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.318969 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-725k5\"" Oct 30 00:38:28.319235 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.318997 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 00:38:28.319235 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.319059 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 00:38:28.340839 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.340804 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm"] Oct 30 00:38:28.432511 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.432475 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/859beab1-e81b-4ff7-bda3-deb4f372ba44-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.432511 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.432526 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t69n\" (UniqueName: \"kubernetes.io/projected/859beab1-e81b-4ff7-bda3-deb4f372ba44-kube-api-access-2t69n\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.432764 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.432657 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/859beab1-e81b-4ff7-bda3-deb4f372ba44-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.432764 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.432696 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/859beab1-e81b-4ff7-bda3-deb4f372ba44-tmp\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.432764 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.432743 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/859beab1-e81b-4ff7-bda3-deb4f372ba44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.432864 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.432766 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/859beab1-e81b-4ff7-bda3-deb4f372ba44-manifests\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.533856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.533762 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/859beab1-e81b-4ff7-bda3-deb4f372ba44-tmp\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.533856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.533814 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/859beab1-e81b-4ff7-bda3-deb4f372ba44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.533856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.533844 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/859beab1-e81b-4ff7-bda3-deb4f372ba44-manifests\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.534111 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.533900 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/859beab1-e81b-4ff7-bda3-deb4f372ba44-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.534111 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.533946 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t69n\" (UniqueName: \"kubernetes.io/projected/859beab1-e81b-4ff7-bda3-deb4f372ba44-kube-api-access-2t69n\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.534111 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.533986 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/859beab1-e81b-4ff7-bda3-deb4f372ba44-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.534373 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.534336 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/859beab1-e81b-4ff7-bda3-deb4f372ba44-manifests\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.536484 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.536449 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/859beab1-e81b-4ff7-bda3-deb4f372ba44-tmp\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.536604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.536490 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/859beab1-e81b-4ff7-bda3-deb4f372ba44-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.536604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.536518 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/859beab1-e81b-4ff7-bda3-deb4f372ba44-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.536604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.536540 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/859beab1-e81b-4ff7-bda3-deb4f372ba44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.578781 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.578745 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t69n\" (UniqueName: \"kubernetes.io/projected/859beab1-e81b-4ff7-bda3-deb4f372ba44-kube-api-access-2t69n\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-rmwfm\" (UID: \"859beab1-e81b-4ff7-bda3-deb4f372ba44\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.606674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.606636 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:28.976511 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:28.976468 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm"] Oct 30 00:38:28.979708 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:38:28.979676 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859beab1_e81b_4ff7_bda3_deb4f372ba44.slice/crio-318b853dfac7fc95ab1e8600833f0e7c961400c989d207b6a02970435cac7d96 WatchSource:0}: Error finding container 318b853dfac7fc95ab1e8600833f0e7c961400c989d207b6a02970435cac7d96: Status 404 returned error can't find the container with id 318b853dfac7fc95ab1e8600833f0e7c961400c989d207b6a02970435cac7d96 Oct 30 00:38:29.695891 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:29.695850 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" event={"ID":"859beab1-e81b-4ff7-bda3-deb4f372ba44","Type":"ContainerStarted","Data":"318b853dfac7fc95ab1e8600833f0e7c961400c989d207b6a02970435cac7d96"} Oct 30 00:38:32.712750 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:32.712711 4927 generic.go:356] "Generic (PLEG): container finished" podID="859beab1-e81b-4ff7-bda3-deb4f372ba44" containerID="f9dc953670a32f9b21190bccbb9c27b0d3c1e29336879eae35336401f75adec3" exitCode=0 Oct 30 00:38:32.713169 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:32.712806 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" event={"ID":"859beab1-e81b-4ff7-bda3-deb4f372ba44","Type":"ContainerDied","Data":"f9dc953670a32f9b21190bccbb9c27b0d3c1e29336879eae35336401f75adec3"} Oct 30 00:38:33.719540 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:33.719501 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" event={"ID":"859beab1-e81b-4ff7-bda3-deb4f372ba44","Type":"ContainerStarted","Data":"cb5bd3bbb5bd8f804016813ba4383e7d5103efe9dc27b183db878e8f88e0780f"} Oct 30 00:38:33.720010 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:33.719598 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:33.803915 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:33.803857 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" podStartSLOduration=2.011073462 podStartE2EDuration="5.803841382s" totalImagesPullingTime="3.292002724s" totalInitContainerRuntime="500.765196ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:33.791417076 +0000 UTC m=+252.363343297" watchObservedRunningTime="2026-10-30 00:38:33.803841382 +0000 UTC m=+252.375767601" Oct 30 00:38:34.440999 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:38:34.440964 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:38:44.726543 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:44.726508 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-rmwfm" Oct 30 00:38:47.988857 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:47.988814 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4"] Oct 30 00:38:47.992556 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:47.992525 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.012426 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.012400 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 00:38:48.012588 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.012544 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 00:38:48.026646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.023881 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sk75l\"" Oct 30 00:38:48.026646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.026384 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4"] Oct 30 00:38:48.085074 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.085035 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b060a4-e781-4a63-b696-98ce6ad20d2f-tls-certs\") pod \"kube-auth-proxy-64d5b5b7d7-t2dm4\" (UID: \"75b060a4-e781-4a63-b696-98ce6ad20d2f\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.085245 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.085087 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqv52\" (UniqueName: \"kubernetes.io/projected/75b060a4-e781-4a63-b696-98ce6ad20d2f-kube-api-access-dqv52\") pod \"kube-auth-proxy-64d5b5b7d7-t2dm4\" (UID: \"75b060a4-e781-4a63-b696-98ce6ad20d2f\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.085245 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.085167 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75b060a4-e781-4a63-b696-98ce6ad20d2f-tmp\") pod \"kube-auth-proxy-64d5b5b7d7-t2dm4\" (UID: \"75b060a4-e781-4a63-b696-98ce6ad20d2f\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.186085 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.186050 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqv52\" (UniqueName: \"kubernetes.io/projected/75b060a4-e781-4a63-b696-98ce6ad20d2f-kube-api-access-dqv52\") pod \"kube-auth-proxy-64d5b5b7d7-t2dm4\" (UID: \"75b060a4-e781-4a63-b696-98ce6ad20d2f\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.186278 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.186094 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75b060a4-e781-4a63-b696-98ce6ad20d2f-tmp\") pod \"kube-auth-proxy-64d5b5b7d7-t2dm4\" (UID: \"75b060a4-e781-4a63-b696-98ce6ad20d2f\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.186278 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.186179 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b060a4-e781-4a63-b696-98ce6ad20d2f-tls-certs\") pod \"kube-auth-proxy-64d5b5b7d7-t2dm4\" (UID: \"75b060a4-e781-4a63-b696-98ce6ad20d2f\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.188529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.188487 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75b060a4-e781-4a63-b696-98ce6ad20d2f-tmp\") pod \"kube-auth-proxy-64d5b5b7d7-t2dm4\" (UID: \"75b060a4-e781-4a63-b696-98ce6ad20d2f\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.188761 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.188738 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b060a4-e781-4a63-b696-98ce6ad20d2f-tls-certs\") pod \"kube-auth-proxy-64d5b5b7d7-t2dm4\" (UID: \"75b060a4-e781-4a63-b696-98ce6ad20d2f\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.208946 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.208913 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqv52\" (UniqueName: \"kubernetes.io/projected/75b060a4-e781-4a63-b696-98ce6ad20d2f-kube-api-access-dqv52\") pod \"kube-auth-proxy-64d5b5b7d7-t2dm4\" (UID: \"75b060a4-e781-4a63-b696-98ce6ad20d2f\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.305375 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.305295 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" Oct 30 00:38:48.610303 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.610222 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4"] Oct 30 00:38:48.613944 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:38:48.613912 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b060a4_e781_4a63_b696_98ce6ad20d2f.slice/crio-47dbe9a1ba94b83c4a164fde2bcc8612cea18de4cac070bd2c48f7bce1f6eabe WatchSource:0}: Error finding container 47dbe9a1ba94b83c4a164fde2bcc8612cea18de4cac070bd2c48f7bce1f6eabe: Status 404 returned error can't find the container with id 47dbe9a1ba94b83c4a164fde2bcc8612cea18de4cac070bd2c48f7bce1f6eabe Oct 30 00:38:48.785116 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:48.785081 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" event={"ID":"75b060a4-e781-4a63-b696-98ce6ad20d2f","Type":"ContainerStarted","Data":"47dbe9a1ba94b83c4a164fde2bcc8612cea18de4cac070bd2c48f7bce1f6eabe"} Oct 30 00:38:49.526292 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.526251 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb"] Oct 30 00:38:49.530773 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.530258 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.552513 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.552473 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 00:38:49.552816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.552735 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 00:38:49.552816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.552791 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 00:38:49.553176 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.553155 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 00:38:49.563272 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.563239 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pqwzr\"" Oct 30 00:38:49.573437 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.573356 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb"] Oct 30 00:38:49.599344 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.599249 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f80a02a-2795-4180-aedb-3a14e1979742-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.599547 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.599508 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldsth\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-kube-api-access-ldsth\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.599650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.599597 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.599650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.599639 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.599720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.599664 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.599765 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.599715 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.599815 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.599779 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.701625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.700886 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f80a02a-2795-4180-aedb-3a14e1979742-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.701625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.700938 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldsth\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-kube-api-access-ldsth\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.701625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.700975 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.701625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.701002 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.701625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.701029 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.701625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.701079 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.701625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.701140 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.702127 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.701791 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.703925 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.703897 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f80a02a-2795-4180-aedb-3a14e1979742-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.704300 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.704275 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.704462 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.704432 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.704787 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.704749 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.726890 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.726853 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.727064 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.726996 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldsth\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-kube-api-access-ldsth\") pod \"istiod-openshift-gateway-7876bfcb74-cbtxb\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:49.847101 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:49.847011 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:50.172795 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:50.172760 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb"] Oct 30 00:38:50.176229 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:38:50.176196 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f80a02a_2795_4180_aedb_3a14e1979742.slice/crio-e5ca8b4670798f0dd4d161dcfcd24075724565aa03ff87c747ba195521fe2117 WatchSource:0}: Error finding container e5ca8b4670798f0dd4d161dcfcd24075724565aa03ff87c747ba195521fe2117: Status 404 returned error can't find the container with id e5ca8b4670798f0dd4d161dcfcd24075724565aa03ff87c747ba195521fe2117 Oct 30 00:38:50.801803 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:50.801760 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" event={"ID":"2f80a02a-2795-4180-aedb-3a14e1979742","Type":"ContainerStarted","Data":"e5ca8b4670798f0dd4d161dcfcd24075724565aa03ff87c747ba195521fe2117"} Oct 30 00:38:51.812100 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:51.812058 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" event={"ID":"75b060a4-e781-4a63-b696-98ce6ad20d2f","Type":"ContainerStarted","Data":"ae751033f8fae5d2483be459b70f9b6a64dd1fb8c78039767a819001e23c7b8a"} Oct 30 00:38:51.852846 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:51.852779 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-t2dm4" podStartSLOduration=2.438060346 podStartE2EDuration="4.852757246s" totalImagesPullingTime="2.4146969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:51.850699051 +0000 UTC m=+270.422625273" watchObservedRunningTime="2026-10-30 00:38:51.852757246 +0000 UTC m=+270.424683466" Oct 30 00:38:52.859440 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:52.859407 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:38:52.859745 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:52.859475 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:38:53.823224 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:53.823185 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" event={"ID":"2f80a02a-2795-4180-aedb-3a14e1979742","Type":"ContainerStarted","Data":"c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49"} Oct 30 00:38:53.823475 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:53.823372 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:53.825275 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:53.825248 4927 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cbtxb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:38:53.825408 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:53.825302 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" podUID="2f80a02a-2795-4180-aedb-3a14e1979742" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:38:53.901828 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:53.901769 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" podStartSLOduration=2.221790844 podStartE2EDuration="4.901753757s" totalImagesPullingTime="2.679962913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:53.896514011 +0000 UTC m=+272.468440232" watchObservedRunningTime="2026-10-30 00:38:53.901753757 +0000 UTC m=+272.473679978" Oct 30 00:38:54.748074 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.748036 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk"] Oct 30 00:38:54.751851 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.751827 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.770955 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.770922 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5bbw2\"" Oct 30 00:38:54.784422 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.784377 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk"] Oct 30 00:38:54.828200 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.828160 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:38:54.850342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.850303 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3316a59b-b981-4db9-a65e-e615e09625fb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.850342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.850339 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnjn\" (UniqueName: \"kubernetes.io/projected/3316a59b-b981-4db9-a65e-e615e09625fb-kube-api-access-hvnjn\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.850600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.850363 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.850600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.850549 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3316a59b-b981-4db9-a65e-e615e09625fb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.850709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.850689 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3316a59b-b981-4db9-a65e-e615e09625fb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.850824 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.850795 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.850930 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.850908 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.850996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.850983 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.851144 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.851128 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.952554 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.952519 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3316a59b-b981-4db9-a65e-e615e09625fb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.952554 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.952565 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3316a59b-b981-4db9-a65e-e615e09625fb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.952608 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.952635 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.952685 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.952714 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.952753 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3316a59b-b981-4db9-a65e-e615e09625fb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.952776 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnjn\" (UniqueName: \"kubernetes.io/projected/3316a59b-b981-4db9-a65e-e615e09625fb-kube-api-access-hvnjn\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.952804 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953393 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.953214 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953393 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.953229 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953393 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.953373 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953493 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.953434 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.953672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.953654 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3316a59b-b981-4db9-a65e-e615e09625fb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.955470 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.955446 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3316a59b-b981-4db9-a65e-e615e09625fb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.955717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.955698 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3316a59b-b981-4db9-a65e-e615e09625fb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.975458 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.975424 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3316a59b-b981-4db9-a65e-e615e09625fb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:54.975648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:54.975534 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnjn\" (UniqueName: \"kubernetes.io/projected/3316a59b-b981-4db9-a65e-e615e09625fb-kube-api-access-hvnjn\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gtglk\" (UID: \"3316a59b-b981-4db9-a65e-e615e09625fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:55.065210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:55.065116 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:38:55.386600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:55.386547 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk"] Oct 30 00:38:55.389747 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:38:55.389718 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3316a59b_b981_4db9_a65e_e615e09625fb.slice/crio-61c83f41558d936a0c4efdc2d6f73c5a7b9a6563d67f3fd5816dc3a71c68a52e WatchSource:0}: Error finding container 61c83f41558d936a0c4efdc2d6f73c5a7b9a6563d67f3fd5816dc3a71c68a52e: Status 404 returned error can't find the container with id 61c83f41558d936a0c4efdc2d6f73c5a7b9a6563d67f3fd5816dc3a71c68a52e Oct 30 00:38:55.832977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:38:55.832877 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" event={"ID":"3316a59b-b981-4db9-a65e-e615e09625fb","Type":"ContainerStarted","Data":"61c83f41558d936a0c4efdc2d6f73c5a7b9a6563d67f3fd5816dc3a71c68a52e"} Oct 30 00:39:03.763756 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.763707 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t"] Oct 30 00:39:03.771195 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.771166 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.785286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.785204 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-ptkmh\"" Oct 30 00:39:03.820007 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.819975 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 00:39:03.822673 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.822641 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t"] Oct 30 00:39:03.834126 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.834087 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/371cb075-558e-48f5-a6a3-5f2c5da8196e-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.834316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.834222 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/371cb075-558e-48f5-a6a3-5f2c5da8196e-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.834316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.834302 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvlz\" (UniqueName: \"kubernetes.io/projected/371cb075-558e-48f5-a6a3-5f2c5da8196e-kube-api-access-wlvlz\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.834441 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.834334 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/371cb075-558e-48f5-a6a3-5f2c5da8196e-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.935052 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.935012 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvlz\" (UniqueName: \"kubernetes.io/projected/371cb075-558e-48f5-a6a3-5f2c5da8196e-kube-api-access-wlvlz\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.935052 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.935054 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/371cb075-558e-48f5-a6a3-5f2c5da8196e-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.935316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.935108 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/371cb075-558e-48f5-a6a3-5f2c5da8196e-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.935316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.935181 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/371cb075-558e-48f5-a6a3-5f2c5da8196e-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.935660 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.935639 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/371cb075-558e-48f5-a6a3-5f2c5da8196e-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.935752 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.935707 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/371cb075-558e-48f5-a6a3-5f2c5da8196e-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.936182 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.936163 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/371cb075-558e-48f5-a6a3-5f2c5da8196e-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:03.958505 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:03.958468 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvlz\" (UniqueName: \"kubernetes.io/projected/371cb075-558e-48f5-a6a3-5f2c5da8196e-kube-api-access-wlvlz\") pod \"kserve-module-controller-manager-5cfb77bf8c-4r98t\" (UID: \"371cb075-558e-48f5-a6a3-5f2c5da8196e\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:04.086897 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:04.086797 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:04.414689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:04.414565 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t"] Oct 30 00:39:04.417712 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:39:04.417632 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371cb075_558e_48f5_a6a3_5f2c5da8196e.slice/crio-63c98240abb9554d379a7717a7dc7bf1917908e103bf2aa6a2d104cb0511cf49 WatchSource:0}: Error finding container 63c98240abb9554d379a7717a7dc7bf1917908e103bf2aa6a2d104cb0511cf49: Status 404 returned error can't find the container with id 63c98240abb9554d379a7717a7dc7bf1917908e103bf2aa6a2d104cb0511cf49 Oct 30 00:39:04.449627 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:39:04.449552 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:39:04.556959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:04.556922 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:39:04.557087 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:04.557004 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:39:04.557087 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:04.557053 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:39:04.875213 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:04.875177 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" event={"ID":"371cb075-558e-48f5-a6a3-5f2c5da8196e","Type":"ContainerStarted","Data":"63c98240abb9554d379a7717a7dc7bf1917908e103bf2aa6a2d104cb0511cf49"} Oct 30 00:39:04.877058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:04.877028 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" event={"ID":"3316a59b-b981-4db9-a65e-e615e09625fb","Type":"ContainerStarted","Data":"eb994cf8888fd2ac997b9dcdf3154f61b4e153b39732739f0dc7ead2d5abb071"} Oct 30 00:39:04.920340 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:04.920282 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" podStartSLOduration=1.756165231 podStartE2EDuration="10.920265357s" totalImagesPullingTime="9.164100126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:04.919425787 +0000 UTC m=+283.491352014" watchObservedRunningTime="2026-10-30 00:39:04.920265357 +0000 UTC m=+283.492191574" Oct 30 00:39:05.065590 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:05.065544 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:39:05.071188 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:05.071160 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:39:05.882747 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:05.882710 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:39:05.883723 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:05.883703 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gtglk" Oct 30 00:39:06.888498 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:06.888453 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" event={"ID":"371cb075-558e-48f5-a6a3-5f2c5da8196e","Type":"ContainerStarted","Data":"8a7205848496458d3e96d3b4fb5049adfa99aa0ea0d1d93e92ba4f0996e43d95"} Oct 30 00:39:06.889035 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:06.888814 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:06.940956 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:06.940895 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" podStartSLOduration=2.401793292 podStartE2EDuration="3.940880044s" totalImagesPullingTime="1.539086752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:06.940866571 +0000 UTC m=+285.512792793" watchObservedRunningTime="2026-10-30 00:39:06.940880044 +0000 UTC m=+285.512806249" Oct 30 00:39:21.600620 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:21.600587 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:39:21.602193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:21.602169 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:39:27.894626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:27.894556 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4r98t" Oct 30 00:39:34.445405 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:39:34.445319 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:39:50.559757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.559714 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs"] Oct 30 00:39:50.570902 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.570864 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.584658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.584547 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs"] Oct 30 00:39:50.647776 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.647733 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.647965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.647794 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e41cfc5-2081-4813-9ac5-a97b47554e53-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.647965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.647868 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhzg\" (UniqueName: \"kubernetes.io/projected/2e41cfc5-2081-4813-9ac5-a97b47554e53-kube-api-access-8zhzg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.647965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.647919 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.647965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.647955 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e41cfc5-2081-4813-9ac5-a97b47554e53-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.648148 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.648014 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.648237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.648214 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.748943 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.748904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.749124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.748955 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e41cfc5-2081-4813-9ac5-a97b47554e53-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.749124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.749069 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhzg\" (UniqueName: \"kubernetes.io/projected/2e41cfc5-2081-4813-9ac5-a97b47554e53-kube-api-access-8zhzg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.749124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.749119 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.749237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.749150 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e41cfc5-2081-4813-9ac5-a97b47554e53-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.749237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.749168 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.749446 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.749425 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.749894 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.749870 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.751445 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.751423 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e41cfc5-2081-4813-9ac5-a97b47554e53-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.751712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.751694 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.751898 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.751876 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e41cfc5-2081-4813-9ac5-a97b47554e53-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.751935 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.751879 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.773211 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.773173 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhzg\" (UniqueName: \"kubernetes.io/projected/2e41cfc5-2081-4813-9ac5-a97b47554e53-kube-api-access-8zhzg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.773626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.773606 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e41cfc5-2081-4813-9ac5-a97b47554e53-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4tcs\" (UID: \"2e41cfc5-2081-4813-9ac5-a97b47554e53\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:50.883016 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:50.882978 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:51.206943 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:51.206851 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs"] Oct 30 00:39:51.210230 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:39:51.210199 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e41cfc5_2081_4813_9ac5_a97b47554e53.slice/crio-1d89cae6ebb0d5a56570cdec99359efe036b54bfa5f34373557dfbe50fba8311 WatchSource:0}: Error finding container 1d89cae6ebb0d5a56570cdec99359efe036b54bfa5f34373557dfbe50fba8311: Status 404 returned error can't find the container with id 1d89cae6ebb0d5a56570cdec99359efe036b54bfa5f34373557dfbe50fba8311 Oct 30 00:39:51.212241 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:51.212203 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:39:51.212330 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:51.212275 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:39:52.117164 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.117122 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" event={"ID":"2e41cfc5-2081-4813-9ac5-a97b47554e53","Type":"ContainerStarted","Data":"aedc7c45a18583ff8b3e1d98b0bfe06ae8837e98c2002caf9579479220404d0b"} Oct 30 00:39:52.117713 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.117179 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" event={"ID":"2e41cfc5-2081-4813-9ac5-a97b47554e53","Type":"ContainerStarted","Data":"1d89cae6ebb0d5a56570cdec99359efe036b54bfa5f34373557dfbe50fba8311"} Oct 30 00:39:52.117713 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.117205 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:52.119016 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.118992 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" Oct 30 00:39:52.183550 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.183471 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4tcs" podStartSLOduration=2.183453527 podStartE2EDuration="2.183453527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:52.170795325 +0000 UTC m=+330.742721546" watchObservedRunningTime="2026-10-30 00:39:52.183453527 +0000 UTC m=+330.755379750" Oct 30 00:39:52.246850 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.246816 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb"] Oct 30 00:39:52.247042 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.247021 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" podUID="2f80a02a-2795-4180-aedb-3a14e1979742" containerName="discovery" containerID="cri-o://c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49" gracePeriod=30 Oct 30 00:39:52.504691 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.504662 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:39:52.568272 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.568238 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-dns-cert\") pod \"2f80a02a-2795-4180-aedb-3a14e1979742\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " Oct 30 00:39:52.568817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.568293 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-kube-api-access-ldsth\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-kube-api-access-ldsth\") pod \"2f80a02a-2795-4180-aedb-3a14e1979742\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " Oct 30 00:39:52.568817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.568316 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-kubeconfig\") pod \"2f80a02a-2795-4180-aedb-3a14e1979742\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " Oct 30 00:39:52.568817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.568346 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f80a02a-2795-4180-aedb-3a14e1979742-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f80a02a-2795-4180-aedb-3a14e1979742-local-certs\") pod \"2f80a02a-2795-4180-aedb-3a14e1979742\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " Oct 30 00:39:52.568817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.568401 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-cacerts\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-cacerts\") pod \"2f80a02a-2795-4180-aedb-3a14e1979742\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " Oct 30 00:39:52.568817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.568464 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-ca-configmap\") pod \"2f80a02a-2795-4180-aedb-3a14e1979742\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " Oct 30 00:39:52.568817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.568495 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-istio-token\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-istio-token\") pod \"2f80a02a-2795-4180-aedb-3a14e1979742\" (UID: \"2f80a02a-2795-4180-aedb-3a14e1979742\") " Oct 30 00:39:52.569734 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.569705 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-ca-configmap" pod "2f80a02a-2795-4180-aedb-3a14e1979742" (UID: "2f80a02a-2795-4180-aedb-3a14e1979742"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:39:52.572678 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.572625 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-istio-token" pod "2f80a02a-2795-4180-aedb-3a14e1979742" (UID: "2f80a02a-2795-4180-aedb-3a14e1979742"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:52.572678 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.572642 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-kubeconfig" pod "2f80a02a-2795-4180-aedb-3a14e1979742" (UID: "2f80a02a-2795-4180-aedb-3a14e1979742"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:52.572678 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.572644 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-kube-api-access-ldsth" pod "2f80a02a-2795-4180-aedb-3a14e1979742" (UID: "2f80a02a-2795-4180-aedb-3a14e1979742"). InnerVolumeSpecName "kube-api-access-ldsth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:52.572880 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.572780 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f80a02a-2795-4180-aedb-3a14e1979742-local-certs" pod "2f80a02a-2795-4180-aedb-3a14e1979742" (UID: "2f80a02a-2795-4180-aedb-3a14e1979742"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:39:52.573879 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.573856 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-dns-cert" pod "2f80a02a-2795-4180-aedb-3a14e1979742" (UID: "2f80a02a-2795-4180-aedb-3a14e1979742"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:52.574061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.574042 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-cacerts" pod "2f80a02a-2795-4180-aedb-3a14e1979742" (UID: "2f80a02a-2795-4180-aedb-3a14e1979742"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:39:52.670307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.670214 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.670307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.670240 4927 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-dns-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.670307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.670251 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldsth\" (UniqueName: \"kubernetes.io/projected/2f80a02a-2795-4180-aedb-3a14e1979742-kube-api-access-ldsth\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.670307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.670260 4927 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-istio-kubeconfig\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.670307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.670269 4927 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f80a02a-2795-4180-aedb-3a14e1979742-local-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.670307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.670277 4927 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f80a02a-2795-4180-aedb-3a14e1979742-cacerts\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:39:52.670307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:52.670284 4927 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f80a02a-2795-4180-aedb-3a14e1979742-istio-csr-ca-configmap\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:39:53.124206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.124176 4927 generic.go:356] "Generic (PLEG): container finished" podID="2f80a02a-2795-4180-aedb-3a14e1979742" containerID="c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49" exitCode=0 Oct 30 00:39:53.124665 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.124247 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" Oct 30 00:39:53.124665 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.124271 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" event={"ID":"2f80a02a-2795-4180-aedb-3a14e1979742","Type":"ContainerDied","Data":"c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49"} Oct 30 00:39:53.124665 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.124327 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb" event={"ID":"2f80a02a-2795-4180-aedb-3a14e1979742","Type":"ContainerDied","Data":"e5ca8b4670798f0dd4d161dcfcd24075724565aa03ff87c747ba195521fe2117"} Oct 30 00:39:53.124665 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.124349 4927 scope.go:122] "RemoveContainer" containerID="c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49" Oct 30 00:39:53.134136 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.134114 4927 scope.go:122] "RemoveContainer" containerID="c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49" Oct 30 00:39:53.134445 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:39:53.134425 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49\": container with ID starting with c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49 not found: ID does not exist" containerID="c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49" Oct 30 00:39:53.134483 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.134456 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49"} err="failed to get container status \"c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49\": rpc error: code = NotFound desc = could not find container \"c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49\": container with ID starting with c2402e84d238f21227dd8f795deb6d956632587ff03c08a91c781e40e20d2d49 not found: ID does not exist" Oct 30 00:39:53.183763 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.183732 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb"] Oct 30 00:39:53.194938 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.194904 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cbtxb"] Oct 30 00:39:53.569496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:39:53.569411 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f80a02a-2795-4180-aedb-3a14e1979742" path="/var/lib/kubelet/pods/2f80a02a-2795-4180-aedb-3a14e1979742/volumes" Oct 30 00:40:04.442349 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:40:04.442291 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:40:34.441714 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:40:34.441675 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:40:43.766716 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.766673 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj"] Oct 30 00:40:43.767336 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.767185 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2f80a02a-2795-4180-aedb-3a14e1979742" containerName="discovery" Oct 30 00:40:43.767336 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.767204 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f80a02a-2795-4180-aedb-3a14e1979742" containerName="discovery" Oct 30 00:40:43.767336 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.767299 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f80a02a-2795-4180-aedb-3a14e1979742" containerName="discovery" Oct 30 00:40:43.770797 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.770774 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.796464 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.796425 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-bgrj2\"" Oct 30 00:40:43.796464 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.796448 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 00:40:43.796464 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.796473 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 00:40:43.798320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.798299 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj"] Oct 30 00:40:43.825222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.825183 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.825433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.825230 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp5ff\" (UniqueName: \"kubernetes.io/projected/35b5834b-3171-4ca8-bf0e-6012f4513462-kube-api-access-mp5ff\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.825433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.825349 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.856564 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.856528 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd"] Oct 30 00:40:43.860291 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.860267 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:43.873903 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.873875 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vmbrg\"" Oct 30 00:40:43.885732 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.885702 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd"] Oct 30 00:40:43.926391 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.926351 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.926627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.926408 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4qv\" (UniqueName: \"kubernetes.io/projected/18b21068-46f6-4335-9658-12285d4ceea4-kube-api-access-nl4qv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:43.926627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.926445 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:43.926627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.926491 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:43.926794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.926644 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.926794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.926691 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp5ff\" (UniqueName: \"kubernetes.io/projected/35b5834b-3171-4ca8-bf0e-6012f4513462-kube-api-access-mp5ff\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.926902 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.926826 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.927062 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.927039 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.950132 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.950059 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp5ff\" (UniqueName: \"kubernetes.io/projected/35b5834b-3171-4ca8-bf0e-6012f4513462-kube-api-access-mp5ff\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:43.959117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.959085 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf"] Oct 30 00:40:43.962887 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.962866 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:43.976313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.976289 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-pl6tf\"" Oct 30 00:40:43.985762 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:43.985721 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf"] Oct 30 00:40:44.027775 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.027688 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.027775 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.027734 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgbx\" (UniqueName: \"kubernetes.io/projected/ef68584e-1ed4-4284-985e-31246e4dd68d-kube-api-access-ftgbx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.027986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.027846 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4qv\" (UniqueName: \"kubernetes.io/projected/18b21068-46f6-4335-9658-12285d4ceea4-kube-api-access-nl4qv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:44.027986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.027902 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:44.027986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.027942 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:44.027986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.027979 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.028329 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.028312 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:44.028366 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.028347 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:44.053489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.053456 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4qv\" (UniqueName: \"kubernetes.io/projected/18b21068-46f6-4335-9658-12285d4ceea4-kube-api-access-nl4qv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:44.060651 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.060559 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8"] Oct 30 00:40:44.064474 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.064448 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.078347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.078320 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-spgqj\"" Oct 30 00:40:44.086936 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.086899 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:44.087206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.087172 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8"] Oct 30 00:40:44.128682 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.128510 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.128682 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.128629 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.129166 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.129076 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fqbh\" (UniqueName: \"kubernetes.io/projected/815f2203-2f1f-4524-9a8a-4bcaa246933b-kube-api-access-2fqbh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.129166 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.129128 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.129298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.129265 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.129355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.129315 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgbx\" (UniqueName: \"kubernetes.io/projected/ef68584e-1ed4-4284-985e-31246e4dd68d-kube-api-access-ftgbx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.129473 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.129446 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.129779 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.129754 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.155750 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.155715 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgbx\" (UniqueName: \"kubernetes.io/projected/ef68584e-1ed4-4284-985e-31246e4dd68d-kube-api-access-ftgbx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.177204 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.177161 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:44.230860 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.230828 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fqbh\" (UniqueName: \"kubernetes.io/projected/815f2203-2f1f-4524-9a8a-4bcaa246933b-kube-api-access-2fqbh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.231007 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.230873 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.231007 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.230948 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.231888 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.231344 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.231888 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.231513 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.257599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.257498 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fqbh\" (UniqueName: \"kubernetes.io/projected/815f2203-2f1f-4524-9a8a-4bcaa246933b-kube-api-access-2fqbh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.280281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.280192 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:44.380940 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.380900 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:44.392378 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.392239 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj"] Oct 30 00:40:44.395219 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:40:44.395186 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b5834b_3171_4ca8_bf0e_6012f4513462.slice/crio-92473678aecfdc86a1018af2dd55343afda4071162a01f196102166c1af2d971 WatchSource:0}: Error finding container 92473678aecfdc86a1018af2dd55343afda4071162a01f196102166c1af2d971: Status 404 returned error can't find the container with id 92473678aecfdc86a1018af2dd55343afda4071162a01f196102166c1af2d971 Oct 30 00:40:44.481181 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.481143 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd"] Oct 30 00:40:44.484207 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:40:44.484177 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b21068_46f6_4335_9658_12285d4ceea4.slice/crio-b4b4ada56d4f8ee1bcfbd9a7178b8338d55a25d4001f7a3f465cf9a8774c7f4a WatchSource:0}: Error finding container b4b4ada56d4f8ee1bcfbd9a7178b8338d55a25d4001f7a3f465cf9a8774c7f4a: Status 404 returned error can't find the container with id b4b4ada56d4f8ee1bcfbd9a7178b8338d55a25d4001f7a3f465cf9a8774c7f4a Oct 30 00:40:44.582291 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.582212 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf"] Oct 30 00:40:44.585023 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:40:44.584990 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef68584e_1ed4_4284_985e_31246e4dd68d.slice/crio-d27fd50d84f811ca16a5e87a568dde3ea3e252d9751726c31a12886025185fa0 WatchSource:0}: Error finding container d27fd50d84f811ca16a5e87a568dde3ea3e252d9751726c31a12886025185fa0: Status 404 returned error can't find the container with id d27fd50d84f811ca16a5e87a568dde3ea3e252d9751726c31a12886025185fa0 Oct 30 00:40:44.686667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:44.686631 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8"] Oct 30 00:40:44.689837 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:40:44.689802 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815f2203_2f1f_4524_9a8a_4bcaa246933b.slice/crio-06138065c2eb3b6d10d541ab56177a39f1c0bd7e46557a4fd4d05058f9a4be36 WatchSource:0}: Error finding container 06138065c2eb3b6d10d541ab56177a39f1c0bd7e46557a4fd4d05058f9a4be36: Status 404 returned error can't find the container with id 06138065c2eb3b6d10d541ab56177a39f1c0bd7e46557a4fd4d05058f9a4be36 Oct 30 00:40:45.368556 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.368515 4927 generic.go:356] "Generic (PLEG): container finished" podID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerID="89f4c0e1dbe934ae697ff6ad0c76fa8f181dd8e3e2dfba6647e44fa73f9b8b3f" exitCode=0 Oct 30 00:40:45.369061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.368599 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" event={"ID":"815f2203-2f1f-4524-9a8a-4bcaa246933b","Type":"ContainerDied","Data":"89f4c0e1dbe934ae697ff6ad0c76fa8f181dd8e3e2dfba6647e44fa73f9b8b3f"} Oct 30 00:40:45.369061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.368654 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" event={"ID":"815f2203-2f1f-4524-9a8a-4bcaa246933b","Type":"ContainerStarted","Data":"06138065c2eb3b6d10d541ab56177a39f1c0bd7e46557a4fd4d05058f9a4be36"} Oct 30 00:40:45.370021 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.370004 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:40:45.370256 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.370231 4927 generic.go:356] "Generic (PLEG): container finished" podID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerID="d21a01a0379f2ccddd36ee1dba655e30b2dfd7e5f6f21539a2dfdeda6a9e784d" exitCode=0 Oct 30 00:40:45.370328 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.370292 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" event={"ID":"35b5834b-3171-4ca8-bf0e-6012f4513462","Type":"ContainerDied","Data":"d21a01a0379f2ccddd36ee1dba655e30b2dfd7e5f6f21539a2dfdeda6a9e784d"} Oct 30 00:40:45.370376 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.370342 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" event={"ID":"35b5834b-3171-4ca8-bf0e-6012f4513462","Type":"ContainerStarted","Data":"92473678aecfdc86a1018af2dd55343afda4071162a01f196102166c1af2d971"} Oct 30 00:40:45.372081 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.372058 4927 generic.go:356] "Generic (PLEG): container finished" podID="18b21068-46f6-4335-9658-12285d4ceea4" containerID="6de6faeb022ee0fdf67d91cbde2ea71e7e4f077a5132c1769598cde6093bcf30" exitCode=0 Oct 30 00:40:45.372180 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.372134 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" event={"ID":"18b21068-46f6-4335-9658-12285d4ceea4","Type":"ContainerDied","Data":"6de6faeb022ee0fdf67d91cbde2ea71e7e4f077a5132c1769598cde6093bcf30"} Oct 30 00:40:45.372245 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.372179 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" event={"ID":"18b21068-46f6-4335-9658-12285d4ceea4","Type":"ContainerStarted","Data":"b4b4ada56d4f8ee1bcfbd9a7178b8338d55a25d4001f7a3f465cf9a8774c7f4a"} Oct 30 00:40:45.373830 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.373809 4927 generic.go:356] "Generic (PLEG): container finished" podID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerID="e284df70da21c93396b688e9000e95973b185b802a3f916c24bb7c35f9a74898" exitCode=0 Oct 30 00:40:45.373917 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.373843 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" event={"ID":"ef68584e-1ed4-4284-985e-31246e4dd68d","Type":"ContainerDied","Data":"e284df70da21c93396b688e9000e95973b185b802a3f916c24bb7c35f9a74898"} Oct 30 00:40:45.373917 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:45.373879 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" event={"ID":"ef68584e-1ed4-4284-985e-31246e4dd68d","Type":"ContainerStarted","Data":"d27fd50d84f811ca16a5e87a568dde3ea3e252d9751726c31a12886025185fa0"} Oct 30 00:40:46.380586 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:46.380548 4927 generic.go:356] "Generic (PLEG): container finished" podID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerID="bb06d57827f0424ba87e1c89721f107541e49c95621dcf1c99b744acb2826e76" exitCode=0 Oct 30 00:40:46.381083 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:46.380639 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" event={"ID":"815f2203-2f1f-4524-9a8a-4bcaa246933b","Type":"ContainerDied","Data":"bb06d57827f0424ba87e1c89721f107541e49c95621dcf1c99b744acb2826e76"} Oct 30 00:40:47.387971 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:47.387935 4927 generic.go:356] "Generic (PLEG): container finished" podID="18b21068-46f6-4335-9658-12285d4ceea4" containerID="eab69f5f9cb9a22e9b05ecb66ea37e904f530ef9b0bac2af81d2f78dcef88ab7" exitCode=0 Oct 30 00:40:47.388407 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:47.388018 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" event={"ID":"18b21068-46f6-4335-9658-12285d4ceea4","Type":"ContainerDied","Data":"eab69f5f9cb9a22e9b05ecb66ea37e904f530ef9b0bac2af81d2f78dcef88ab7"} Oct 30 00:40:47.390377 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:47.390351 4927 generic.go:356] "Generic (PLEG): container finished" podID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerID="4c1c4be32b9a0e3bf56b42ff032b4c8b80aefdff4bca7998c416485310f2cd1a" exitCode=0 Oct 30 00:40:47.390531 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:47.390436 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" event={"ID":"ef68584e-1ed4-4284-985e-31246e4dd68d","Type":"ContainerDied","Data":"4c1c4be32b9a0e3bf56b42ff032b4c8b80aefdff4bca7998c416485310f2cd1a"} Oct 30 00:40:47.393689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:47.393426 4927 generic.go:356] "Generic (PLEG): container finished" podID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerID="02c7c60360620d0d9ae488c49101b6589b49685e2e9952e26e2bc7853eeeb335" exitCode=0 Oct 30 00:40:47.393689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:47.393649 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" event={"ID":"815f2203-2f1f-4524-9a8a-4bcaa246933b","Type":"ContainerDied","Data":"02c7c60360620d0d9ae488c49101b6589b49685e2e9952e26e2bc7853eeeb335"} Oct 30 00:40:47.395819 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:47.395798 4927 generic.go:356] "Generic (PLEG): container finished" podID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerID="cb2da23bc918cbc47df60b3e80dadda5239cca352ecc29acf1a01e32d53e28de" exitCode=0 Oct 30 00:40:47.395915 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:47.395839 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" event={"ID":"35b5834b-3171-4ca8-bf0e-6012f4513462","Type":"ContainerDied","Data":"cb2da23bc918cbc47df60b3e80dadda5239cca352ecc29acf1a01e32d53e28de"} Oct 30 00:40:48.404037 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.404003 4927 generic.go:356] "Generic (PLEG): container finished" podID="18b21068-46f6-4335-9658-12285d4ceea4" containerID="16ce2521d987cab6a309371078822e9cfc279a76a619979fd4ee573793a37731" exitCode=0 Oct 30 00:40:48.404470 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.404100 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" event={"ID":"18b21068-46f6-4335-9658-12285d4ceea4","Type":"ContainerDied","Data":"16ce2521d987cab6a309371078822e9cfc279a76a619979fd4ee573793a37731"} Oct 30 00:40:48.406272 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.406241 4927 generic.go:356] "Generic (PLEG): container finished" podID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerID="a81efb073d0ac128067ae85eaf4747bd4f70ecbdb53a011eca0811d9a6dc8323" exitCode=0 Oct 30 00:40:48.406412 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.406330 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" event={"ID":"ef68584e-1ed4-4284-985e-31246e4dd68d","Type":"ContainerDied","Data":"a81efb073d0ac128067ae85eaf4747bd4f70ecbdb53a011eca0811d9a6dc8323"} Oct 30 00:40:48.408389 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.408366 4927 generic.go:356] "Generic (PLEG): container finished" podID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerID="a2686d10f012041b41d31cc274e289cb21c754333cad43f9c0d282de83902f32" exitCode=0 Oct 30 00:40:48.408467 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.408434 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" event={"ID":"35b5834b-3171-4ca8-bf0e-6012f4513462","Type":"ContainerDied","Data":"a2686d10f012041b41d31cc274e289cb21c754333cad43f9c0d282de83902f32"} Oct 30 00:40:48.544334 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.544305 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:48.677744 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.677650 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-util\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-util\") pod \"815f2203-2f1f-4524-9a8a-4bcaa246933b\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " Oct 30 00:40:48.677902 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.677768 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-bundle\") pod \"815f2203-2f1f-4524-9a8a-4bcaa246933b\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " Oct 30 00:40:48.677902 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.677834 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815f2203-2f1f-4524-9a8a-4bcaa246933b-kube-api-access-2fqbh\" (UniqueName: \"kubernetes.io/projected/815f2203-2f1f-4524-9a8a-4bcaa246933b-kube-api-access-2fqbh\") pod \"815f2203-2f1f-4524-9a8a-4bcaa246933b\" (UID: \"815f2203-2f1f-4524-9a8a-4bcaa246933b\") " Oct 30 00:40:48.678288 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.678259 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-bundle" pod "815f2203-2f1f-4524-9a8a-4bcaa246933b" (UID: "815f2203-2f1f-4524-9a8a-4bcaa246933b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:48.680212 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.680188 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815f2203-2f1f-4524-9a8a-4bcaa246933b-kube-api-access-2fqbh" pod "815f2203-2f1f-4524-9a8a-4bcaa246933b" (UID: "815f2203-2f1f-4524-9a8a-4bcaa246933b"). InnerVolumeSpecName "kube-api-access-2fqbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:48.683256 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.683218 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-util" pod "815f2203-2f1f-4524-9a8a-4bcaa246933b" (UID: "815f2203-2f1f-4524-9a8a-4bcaa246933b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:48.779474 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.779431 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fqbh\" (UniqueName: \"kubernetes.io/projected/815f2203-2f1f-4524-9a8a-4bcaa246933b-kube-api-access-2fqbh\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:48.779474 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.779467 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-util\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:48.779474 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:48.779477 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815f2203-2f1f-4524-9a8a-4bcaa246933b-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:49.415525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.415483 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" event={"ID":"815f2203-2f1f-4524-9a8a-4bcaa246933b","Type":"ContainerDied","Data":"06138065c2eb3b6d10d541ab56177a39f1c0bd7e46557a4fd4d05058f9a4be36"} Oct 30 00:40:49.415525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.415517 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wlgj8" Oct 30 00:40:49.416115 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.415543 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06138065c2eb3b6d10d541ab56177a39f1c0bd7e46557a4fd4d05058f9a4be36" Oct 30 00:40:49.564047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.564008 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:49.598630 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.598604 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:49.601715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.601686 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:49.690061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.689958 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-bundle\") pod \"35b5834b-3171-4ca8-bf0e-6012f4513462\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " Oct 30 00:40:49.690061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.690005 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b21068-46f6-4335-9658-12285d4ceea4-kube-api-access-nl4qv\" (UniqueName: \"kubernetes.io/projected/18b21068-46f6-4335-9658-12285d4ceea4-kube-api-access-nl4qv\") pod \"18b21068-46f6-4335-9658-12285d4ceea4\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " Oct 30 00:40:49.690061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.690039 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-util\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-util\") pod \"35b5834b-3171-4ca8-bf0e-6012f4513462\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " Oct 30 00:40:49.690326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.690093 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-util\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-util\") pod \"18b21068-46f6-4335-9658-12285d4ceea4\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " Oct 30 00:40:49.690326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.690135 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-util\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-util\") pod \"ef68584e-1ed4-4284-985e-31246e4dd68d\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " Oct 30 00:40:49.690326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.690195 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b5834b-3171-4ca8-bf0e-6012f4513462-kube-api-access-mp5ff\" (UniqueName: \"kubernetes.io/projected/35b5834b-3171-4ca8-bf0e-6012f4513462-kube-api-access-mp5ff\") pod \"35b5834b-3171-4ca8-bf0e-6012f4513462\" (UID: \"35b5834b-3171-4ca8-bf0e-6012f4513462\") " Oct 30 00:40:49.690558 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.690535 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-bundle\") pod \"ef68584e-1ed4-4284-985e-31246e4dd68d\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " Oct 30 00:40:49.690719 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.690703 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-bundle\") pod \"18b21068-46f6-4335-9658-12285d4ceea4\" (UID: \"18b21068-46f6-4335-9658-12285d4ceea4\") " Oct 30 00:40:49.690871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.690852 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef68584e-1ed4-4284-985e-31246e4dd68d-kube-api-access-ftgbx\" (UniqueName: \"kubernetes.io/projected/ef68584e-1ed4-4284-985e-31246e4dd68d-kube-api-access-ftgbx\") pod \"ef68584e-1ed4-4284-985e-31246e4dd68d\" (UID: \"ef68584e-1ed4-4284-985e-31246e4dd68d\") " Oct 30 00:40:49.691250 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.690841 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-bundle" pod "ef68584e-1ed4-4284-985e-31246e4dd68d" (UID: "ef68584e-1ed4-4284-985e-31246e4dd68d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:49.691374 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.691007 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-bundle" pod "35b5834b-3171-4ca8-bf0e-6012f4513462" (UID: "35b5834b-3171-4ca8-bf0e-6012f4513462"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:49.691641 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.691601 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-bundle" pod "18b21068-46f6-4335-9658-12285d4ceea4" (UID: "18b21068-46f6-4335-9658-12285d4ceea4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:49.692627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.692601 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b5834b-3171-4ca8-bf0e-6012f4513462-kube-api-access-mp5ff" pod "35b5834b-3171-4ca8-bf0e-6012f4513462" (UID: "35b5834b-3171-4ca8-bf0e-6012f4513462"). InnerVolumeSpecName "kube-api-access-mp5ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:49.692840 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.692819 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b21068-46f6-4335-9658-12285d4ceea4-kube-api-access-nl4qv" pod "18b21068-46f6-4335-9658-12285d4ceea4" (UID: "18b21068-46f6-4335-9658-12285d4ceea4"). InnerVolumeSpecName "kube-api-access-nl4qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:49.693212 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.693194 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef68584e-1ed4-4284-985e-31246e4dd68d-kube-api-access-ftgbx" pod "ef68584e-1ed4-4284-985e-31246e4dd68d" (UID: "ef68584e-1ed4-4284-985e-31246e4dd68d"). InnerVolumeSpecName "kube-api-access-ftgbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:49.695611 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.695559 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-util" pod "ef68584e-1ed4-4284-985e-31246e4dd68d" (UID: "ef68584e-1ed4-4284-985e-31246e4dd68d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:49.699268 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.699239 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-util" pod "35b5834b-3171-4ca8-bf0e-6012f4513462" (UID: "35b5834b-3171-4ca8-bf0e-6012f4513462"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:49.699383 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.699237 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-util" pod "18b21068-46f6-4335-9658-12285d4ceea4" (UID: "18b21068-46f6-4335-9658-12285d4ceea4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:40:49.792155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.792116 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:49.792155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.792148 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl4qv\" (UniqueName: \"kubernetes.io/projected/18b21068-46f6-4335-9658-12285d4ceea4-kube-api-access-nl4qv\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:49.792155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.792159 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35b5834b-3171-4ca8-bf0e-6012f4513462-util\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:49.792155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.792167 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-util\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:49.792419 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.792176 4927 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-util\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:49.792419 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.792184 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp5ff\" (UniqueName: \"kubernetes.io/projected/35b5834b-3171-4ca8-bf0e-6012f4513462-kube-api-access-mp5ff\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:49.792419 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.792192 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef68584e-1ed4-4284-985e-31246e4dd68d-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:49.792419 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.792200 4927 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b21068-46f6-4335-9658-12285d4ceea4-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:49.792419 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:49.792209 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgbx\" (UniqueName: \"kubernetes.io/projected/ef68584e-1ed4-4284-985e-31246e4dd68d-kube-api-access-ftgbx\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:40:50.421868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:50.421837 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" Oct 30 00:40:50.421868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:50.421848 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7qpj" event={"ID":"35b5834b-3171-4ca8-bf0e-6012f4513462","Type":"ContainerDied","Data":"92473678aecfdc86a1018af2dd55343afda4071162a01f196102166c1af2d971"} Oct 30 00:40:50.422413 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:50.421897 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92473678aecfdc86a1018af2dd55343afda4071162a01f196102166c1af2d971" Oct 30 00:40:50.423929 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:50.423898 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" Oct 30 00:40:50.423929 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:50.423906 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axwlbd" event={"ID":"18b21068-46f6-4335-9658-12285d4ceea4","Type":"ContainerDied","Data":"b4b4ada56d4f8ee1bcfbd9a7178b8338d55a25d4001f7a3f465cf9a8774c7f4a"} Oct 30 00:40:50.424105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:50.423956 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b4ada56d4f8ee1bcfbd9a7178b8338d55a25d4001f7a3f465cf9a8774c7f4a" Oct 30 00:40:50.425884 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:50.425860 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" event={"ID":"ef68584e-1ed4-4284-985e-31246e4dd68d","Type":"ContainerDied","Data":"d27fd50d84f811ca16a5e87a568dde3ea3e252d9751726c31a12886025185fa0"} Oct 30 00:40:50.425884 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:50.425877 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67kslf" Oct 30 00:40:50.426036 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:50.425902 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d27fd50d84f811ca16a5e87a568dde3ea3e252d9751726c31a12886025185fa0" Oct 30 00:40:54.711876 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.711838 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc"] Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712230 4927 cpu_manager.go:414] "containerMap: removing container" podUID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerName="util" Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712243 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerName="util" Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712252 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerName="pull" Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712257 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerName="pull" Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712269 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerName="util" Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712274 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerName="util" Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712283 4927 cpu_manager.go:414] "containerMap: removing container" podUID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerName="pull" Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712288 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerName="pull" Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712307 4927 cpu_manager.go:414] "containerMap: removing container" podUID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerName="extract" Oct 30 00:40:54.712307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712312 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712322 4927 cpu_manager.go:414] "containerMap: removing container" podUID="18b21068-46f6-4335-9658-12285d4ceea4" containerName="pull" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712328 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b21068-46f6-4335-9658-12285d4ceea4" containerName="pull" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712334 4927 cpu_manager.go:414] "containerMap: removing container" podUID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerName="pull" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712338 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerName="pull" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712350 4927 cpu_manager.go:414] "containerMap: removing container" podUID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerName="util" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712355 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerName="util" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712362 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712366 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712374 4927 cpu_manager.go:414] "containerMap: removing container" podUID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712381 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712393 4927 cpu_manager.go:414] "containerMap: removing container" podUID="18b21068-46f6-4335-9658-12285d4ceea4" containerName="util" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712398 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b21068-46f6-4335-9658-12285d4ceea4" containerName="util" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712405 4927 cpu_manager.go:414] "containerMap: removing container" podUID="18b21068-46f6-4335-9658-12285d4ceea4" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712409 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b21068-46f6-4335-9658-12285d4ceea4" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712470 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="815f2203-2f1f-4524-9a8a-4bcaa246933b" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712477 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b5834b-3171-4ca8-bf0e-6012f4513462" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712482 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef68584e-1ed4-4284-985e-31246e4dd68d" containerName="extract" Oct 30 00:40:54.712616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.712491 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b21068-46f6-4335-9658-12285d4ceea4" containerName="extract" Oct 30 00:40:54.717029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.717007 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" Oct 30 00:40:54.744409 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.744380 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 00:40:54.744635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.744381 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 00:40:54.744635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.744387 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qntb9\"" Oct 30 00:40:54.744635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.744404 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 00:40:54.746970 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.746945 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc"] Oct 30 00:40:54.842540 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.842501 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4tr\" (UniqueName: \"kubernetes.io/projected/866dff6e-c8f0-4d93-97ee-f6fb9dacdb46-kube-api-access-zf4tr\") pod \"dns-operator-controller-manager-65bfd79598-lw8mc\" (UID: \"866dff6e-c8f0-4d93-97ee-f6fb9dacdb46\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" Oct 30 00:40:54.943150 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.943100 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4tr\" (UniqueName: \"kubernetes.io/projected/866dff6e-c8f0-4d93-97ee-f6fb9dacdb46-kube-api-access-zf4tr\") pod \"dns-operator-controller-manager-65bfd79598-lw8mc\" (UID: \"866dff6e-c8f0-4d93-97ee-f6fb9dacdb46\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" Oct 30 00:40:54.998632 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:54.998528 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4tr\" (UniqueName: \"kubernetes.io/projected/866dff6e-c8f0-4d93-97ee-f6fb9dacdb46-kube-api-access-zf4tr\") pod \"dns-operator-controller-manager-65bfd79598-lw8mc\" (UID: \"866dff6e-c8f0-4d93-97ee-f6fb9dacdb46\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" Oct 30 00:40:55.029711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:55.029670 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" Oct 30 00:40:55.342545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:55.342517 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc"] Oct 30 00:40:55.345612 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:40:55.345555 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866dff6e_c8f0_4d93_97ee_f6fb9dacdb46.slice/crio-331edfc30c2f1fd8e20735d4758cb5237d60b48ad4994ca30ae27b65723d7c86 WatchSource:0}: Error finding container 331edfc30c2f1fd8e20735d4758cb5237d60b48ad4994ca30ae27b65723d7c86: Status 404 returned error can't find the container with id 331edfc30c2f1fd8e20735d4758cb5237d60b48ad4994ca30ae27b65723d7c86 Oct 30 00:40:55.452036 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:55.451993 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" event={"ID":"866dff6e-c8f0-4d93-97ee-f6fb9dacdb46","Type":"ContainerStarted","Data":"331edfc30c2f1fd8e20735d4758cb5237d60b48ad4994ca30ae27b65723d7c86"} Oct 30 00:40:58.470684 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:58.470566 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" event={"ID":"866dff6e-c8f0-4d93-97ee-f6fb9dacdb46","Type":"ContainerStarted","Data":"7627ab3d6195f36b84269802ecc54a4b982f5cb9e647ccc4ed0631b2f1ff24ac"} Oct 30 00:40:58.471074 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:58.470738 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" Oct 30 00:40:58.510263 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:58.510194 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" podStartSLOduration=1.652030111 podStartE2EDuration="4.510178209s" totalImagesPullingTime="2.858148098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:58.509794926 +0000 UTC m=+397.081721174" watchObservedRunningTime="2026-10-30 00:40:58.510178209 +0000 UTC m=+397.082104432" Oct 30 00:40:59.063938 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.063896 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp"] Oct 30 00:40:59.067610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.067563 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:40:59.100920 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.100886 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-87vj2\"" Oct 30 00:40:59.115203 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.115169 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp"] Oct 30 00:40:59.182219 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.182181 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" (UID: \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:40:59.182416 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.182256 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk87f\" (UniqueName: \"kubernetes.io/projected/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-kube-api-access-zk87f\") pod \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" (UID: \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:40:59.283286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.283240 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" (UID: \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:40:59.283494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.283330 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk87f\" (UniqueName: \"kubernetes.io/projected/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-kube-api-access-zk87f\") pod \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" (UID: \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:40:59.283812 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.283789 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" (UID: \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:40:59.348718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.348646 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk87f\" (UniqueName: \"kubernetes.io/projected/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-kube-api-access-zk87f\") pod \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" (UID: \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:40:59.380514 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.380471 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:40:59.729320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:40:59.729284 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp"] Oct 30 00:40:59.733035 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:40:59.732996 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f83e6a4_07f9_491b_a19a_dfc9f61a5555.slice/crio-c5b85e9a3410f49ec7e0895ed2e3c0a73b8516efd793c4aaf3bf57cb42b85189 WatchSource:0}: Error finding container c5b85e9a3410f49ec7e0895ed2e3c0a73b8516efd793c4aaf3bf57cb42b85189: Status 404 returned error can't find the container with id c5b85e9a3410f49ec7e0895ed2e3c0a73b8516efd793c4aaf3bf57cb42b85189 Oct 30 00:41:00.483373 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:00.483333 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" event={"ID":"0f83e6a4-07f9-491b-a19a-dfc9f61a5555","Type":"ContainerStarted","Data":"c5b85e9a3410f49ec7e0895ed2e3c0a73b8516efd793c4aaf3bf57cb42b85189"} Oct 30 00:41:03.501834 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:03.501798 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" event={"ID":"0f83e6a4-07f9-491b-a19a-dfc9f61a5555","Type":"ContainerStarted","Data":"b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89"} Oct 30 00:41:03.502291 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:03.501906 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:03.567123 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:03.567066 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" podStartSLOduration=0.917385019 podStartE2EDuration="4.567040302s" totalImagesPullingTime="3.649655283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:03.553747798 +0000 UTC m=+402.125674023" watchObservedRunningTime="2026-10-30 00:41:03.567040302 +0000 UTC m=+402.138966526" Oct 30 00:41:04.011883 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:04.011849 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn"] Oct 30 00:41:04.015276 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:04.015254 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:04.024779 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:04.024753 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88kl\" (UniqueName: \"kubernetes.io/projected/bf390a68-091c-4f0a-ad56-b8572d50117d-kube-api-access-h88kl\") pod \"limitador-operator-controller-manager-856dd584c5-w66nn\" (UID: \"bf390a68-091c-4f0a-ad56-b8572d50117d\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:04.043478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:04.043453 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zz8br\"" Oct 30 00:41:04.045946 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:04.045923 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn"] Oct 30 00:41:04.126118 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:04.126081 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88kl\" (UniqueName: \"kubernetes.io/projected/bf390a68-091c-4f0a-ad56-b8572d50117d-kube-api-access-h88kl\") pod \"limitador-operator-controller-manager-856dd584c5-w66nn\" (UID: \"bf390a68-091c-4f0a-ad56-b8572d50117d\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:04.148880 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:04.148851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88kl\" (UniqueName: \"kubernetes.io/projected/bf390a68-091c-4f0a-ad56-b8572d50117d-kube-api-access-h88kl\") pod \"limitador-operator-controller-manager-856dd584c5-w66nn\" (UID: \"bf390a68-091c-4f0a-ad56-b8572d50117d\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:04.327744 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:04.327640 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:04.440801 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:04.440763 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:41:04.628662 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:04.628501 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn"] Oct 30 00:41:04.631339 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:41:04.631304 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf390a68_091c_4f0a_ad56_b8572d50117d.slice/crio-4b691f1aa52b023d31a14ed2cb33efb11ad146dc4650b249f6f39e24c1aad676 WatchSource:0}: Error finding container 4b691f1aa52b023d31a14ed2cb33efb11ad146dc4650b249f6f39e24c1aad676: Status 404 returned error can't find the container with id 4b691f1aa52b023d31a14ed2cb33efb11ad146dc4650b249f6f39e24c1aad676 Oct 30 00:41:05.513964 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:05.513922 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" event={"ID":"bf390a68-091c-4f0a-ad56-b8572d50117d","Type":"ContainerStarted","Data":"4b691f1aa52b023d31a14ed2cb33efb11ad146dc4650b249f6f39e24c1aad676"} Oct 30 00:41:06.521535 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:06.521449 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" event={"ID":"bf390a68-091c-4f0a-ad56-b8572d50117d","Type":"ContainerStarted","Data":"24802c4b138f2b463b83fc6319f05e8177ca461cb095f40f9343f93ef378523c"} Oct 30 00:41:06.521955 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:06.521605 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:06.567770 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:06.567713 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" podStartSLOduration=2.0626665 podStartE2EDuration="3.567696367s" totalImagesPullingTime="1.505029867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:06.566497964 +0000 UTC m=+405.138424185" watchObservedRunningTime="2026-10-30 00:41:06.567696367 +0000 UTC m=+405.139622591" Oct 30 00:41:09.478008 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:09.477973 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-lw8mc" Oct 30 00:41:14.509524 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:14.509493 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:17.529461 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:17.529429 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:18.247327 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.247286 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp"] Oct 30 00:41:18.249452 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.249413 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" containerName="manager" containerID="cri-o://b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89" gracePeriod=2 Oct 30 00:41:18.249879 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.249850 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp"] Oct 30 00:41:18.258323 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.258295 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8"] Oct 30 00:41:18.259335 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.258858 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" containerName="manager" Oct 30 00:41:18.259335 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.258882 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" containerName="manager" Oct 30 00:41:18.259335 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.258970 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" containerName="manager" Oct 30 00:41:18.262521 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.262491 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:18.264052 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.264034 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:18.276770 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.276727 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:18.288871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.288833 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8"] Oct 30 00:41:18.334985 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.334961 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn"] Oct 30 00:41:18.335168 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.335142 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" podUID="bf390a68-091c-4f0a-ad56-b8572d50117d" containerName="manager" containerID="cri-o://24802c4b138f2b463b83fc6319f05e8177ca461cb095f40f9343f93ef378523c" gracePeriod=2 Oct 30 00:41:18.344446 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.344416 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn"] Oct 30 00:41:18.347859 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.347828 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:18.358062 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.358034 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5b5q\" (UniqueName: \"kubernetes.io/projected/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-kube-api-access-n5b5q\") pod \"kuadrant-operator-controller-manager-6b6fb56846-n4qh8\" (UID: \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:18.358193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.358157 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-n4qh8\" (UID: \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:18.360399 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.360362 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w66nn\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="bf390a68-091c-4f0a-ad56-b8572d50117d" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:18.368793 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.368761 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm"] Oct 30 00:41:18.369375 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.369354 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bf390a68-091c-4f0a-ad56-b8572d50117d" containerName="manager" Oct 30 00:41:18.369439 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.369380 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf390a68-091c-4f0a-ad56-b8572d50117d" containerName="manager" Oct 30 00:41:18.369503 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.369492 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf390a68-091c-4f0a-ad56-b8572d50117d" containerName="manager" Oct 30 00:41:18.373054 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.373035 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" Oct 30 00:41:18.385944 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.385906 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w66nn\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="bf390a68-091c-4f0a-ad56-b8572d50117d" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:18.391757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.391730 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm"] Oct 30 00:41:18.438735 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.438688 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:18.458869 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.458835 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5b5q\" (UniqueName: \"kubernetes.io/projected/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-kube-api-access-n5b5q\") pod \"kuadrant-operator-controller-manager-6b6fb56846-n4qh8\" (UID: \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:18.459035 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.458919 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txz8\" (UniqueName: \"kubernetes.io/projected/52eb857f-d1e0-41a2-95bc-08c5c9d992fb-kube-api-access-9txz8\") pod \"limitador-operator-controller-manager-856dd584c5-wjgtm\" (UID: \"52eb857f-d1e0-41a2-95bc-08c5c9d992fb\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" Oct 30 00:41:18.459035 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.458944 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-n4qh8\" (UID: \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:18.459465 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.459432 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-n4qh8\" (UID: \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:18.481669 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.481614 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5b5q\" (UniqueName: \"kubernetes.io/projected/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-kube-api-access-n5b5q\") pod \"kuadrant-operator-controller-manager-6b6fb56846-n4qh8\" (UID: \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:18.560229 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.560194 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txz8\" (UniqueName: \"kubernetes.io/projected/52eb857f-d1e0-41a2-95bc-08c5c9d992fb-kube-api-access-9txz8\") pod \"limitador-operator-controller-manager-856dd584c5-wjgtm\" (UID: \"52eb857f-d1e0-41a2-95bc-08c5c9d992fb\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" Oct 30 00:41:18.583397 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.583374 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txz8\" (UniqueName: \"kubernetes.io/projected/52eb857f-d1e0-41a2-95bc-08c5c9d992fb-kube-api-access-9txz8\") pod \"limitador-operator-controller-manager-856dd584c5-wjgtm\" (UID: \"52eb857f-d1e0-41a2-95bc-08c5c9d992fb\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" Oct 30 00:41:18.583660 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.583643 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:18.584527 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.584501 4927 generic.go:356] "Generic (PLEG): container finished" podID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" containerID="b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89" exitCode=0 Oct 30 00:41:18.584654 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.584602 4927 scope.go:122] "RemoveContainer" containerID="b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89" Oct 30 00:41:18.585932 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.585915 4927 generic.go:356] "Generic (PLEG): container finished" podID="bf390a68-091c-4f0a-ad56-b8572d50117d" containerID="24802c4b138f2b463b83fc6319f05e8177ca461cb095f40f9343f93ef378523c" exitCode=0 Oct 30 00:41:18.586648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.586632 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:18.592875 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.592858 4927 scope.go:122] "RemoveContainer" containerID="b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89" Oct 30 00:41:18.593171 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.593147 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89\": container with ID starting with b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89 not found: ID does not exist" containerID="b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89" Oct 30 00:41:18.593219 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.593183 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89"} err="failed to get container status \"b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89\": rpc error: code = NotFound desc = could not find container \"b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89\": container with ID starting with b46445f0cd1b6c8d614390b96fced14f437281b5eae01ca3884aebd7f0948d89 not found: ID does not exist" Oct 30 00:41:18.596397 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.596374 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:18.609081 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.609052 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w66nn\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="bf390a68-091c-4f0a-ad56-b8572d50117d" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:18.621743 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.621709 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-djplp\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:18.634369 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:18.634337 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w66nn\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="bf390a68-091c-4f0a-ad56-b8572d50117d" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:18.639533 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.639513 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:18.660902 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.660876 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-kube-api-access-zk87f\" (UniqueName: \"kubernetes.io/projected/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-kube-api-access-zk87f\") pod \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\" (UID: \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\") " Oct 30 00:41:18.661014 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.660961 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf390a68-091c-4f0a-ad56-b8572d50117d-kube-api-access-h88kl\" (UniqueName: \"kubernetes.io/projected/bf390a68-091c-4f0a-ad56-b8572d50117d-kube-api-access-h88kl\") pod \"bf390a68-091c-4f0a-ad56-b8572d50117d\" (UID: \"bf390a68-091c-4f0a-ad56-b8572d50117d\") " Oct 30 00:41:18.661052 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.661025 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-extensions-socket-volume\") pod \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\" (UID: \"0f83e6a4-07f9-491b-a19a-dfc9f61a5555\") " Oct 30 00:41:18.661524 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.661498 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-extensions-socket-volume" pod "0f83e6a4-07f9-491b-a19a-dfc9f61a5555" (UID: "0f83e6a4-07f9-491b-a19a-dfc9f61a5555"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:18.663168 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.663138 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf390a68-091c-4f0a-ad56-b8572d50117d-kube-api-access-h88kl" pod "bf390a68-091c-4f0a-ad56-b8572d50117d" (UID: "bf390a68-091c-4f0a-ad56-b8572d50117d"). InnerVolumeSpecName "kube-api-access-h88kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:18.663263 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.663183 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-kube-api-access-zk87f" pod "0f83e6a4-07f9-491b-a19a-dfc9f61a5555" (UID: "0f83e6a4-07f9-491b-a19a-dfc9f61a5555"). InnerVolumeSpecName "kube-api-access-zk87f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:18.722626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.722588 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" Oct 30 00:41:18.762689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.762609 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk87f\" (UniqueName: \"kubernetes.io/projected/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-kube-api-access-zk87f\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:41:18.762689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.762641 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h88kl\" (UniqueName: \"kubernetes.io/projected/bf390a68-091c-4f0a-ad56-b8572d50117d-kube-api-access-h88kl\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:41:18.762689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.762651 4927 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f83e6a4-07f9-491b-a19a-dfc9f61a5555-extensions-socket-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:41:18.972864 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:18.972837 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8"] Oct 30 00:41:18.975318 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:41:18.975280 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89cd976_4ca1_4316_b316_8b5d2c13b6ca.slice/crio-8922b6fd71a9036af086c0582bf3e706499f56b1e26c104e0a77de804b385f81 WatchSource:0}: Error finding container 8922b6fd71a9036af086c0582bf3e706499f56b1e26c104e0a77de804b385f81: Status 404 returned error can't find the container with id 8922b6fd71a9036af086c0582bf3e706499f56b1e26c104e0a77de804b385f81 Oct 30 00:41:19.032941 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.032911 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm"] Oct 30 00:41:19.035744 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:41:19.035708 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52eb857f_d1e0_41a2_95bc_08c5c9d992fb.slice/crio-b4c6d3f1107cd92ab4b3e8b40c5f553ac2cd147f2a8f1aeab46b1a0edb1af54e WatchSource:0}: Error finding container b4c6d3f1107cd92ab4b3e8b40c5f553ac2cd147f2a8f1aeab46b1a0edb1af54e: Status 404 returned error can't find the container with id b4c6d3f1107cd92ab4b3e8b40c5f553ac2cd147f2a8f1aeab46b1a0edb1af54e Oct 30 00:41:19.569876 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.569791 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f83e6a4-07f9-491b-a19a-dfc9f61a5555" path="/var/lib/kubelet/pods/0f83e6a4-07f9-491b-a19a-dfc9f61a5555/volumes" Oct 30 00:41:19.570300 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.570267 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf390a68-091c-4f0a-ad56-b8572d50117d" path="/var/lib/kubelet/pods/bf390a68-091c-4f0a-ad56-b8572d50117d/volumes" Oct 30 00:41:19.592810 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.592774 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" event={"ID":"f89cd976-4ca1-4316-b316-8b5d2c13b6ca","Type":"ContainerStarted","Data":"835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef"} Oct 30 00:41:19.592973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.592834 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" event={"ID":"f89cd976-4ca1-4316-b316-8b5d2c13b6ca","Type":"ContainerStarted","Data":"8922b6fd71a9036af086c0582bf3e706499f56b1e26c104e0a77de804b385f81"} Oct 30 00:41:19.592973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.592957 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:19.594491 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.594469 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" event={"ID":"52eb857f-d1e0-41a2-95bc-08c5c9d992fb","Type":"ContainerStarted","Data":"94c81eee751f0755528f77f8b3b9077cd4a35abcdf27f45c8e6f801740a0f85a"} Oct 30 00:41:19.594634 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.594511 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" event={"ID":"52eb857f-d1e0-41a2-95bc-08c5c9d992fb","Type":"ContainerStarted","Data":"b4c6d3f1107cd92ab4b3e8b40c5f553ac2cd147f2a8f1aeab46b1a0edb1af54e"} Oct 30 00:41:19.594634 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.594535 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" Oct 30 00:41:19.595837 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.595818 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w66nn" Oct 30 00:41:19.595936 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.595864 4927 scope.go:122] "RemoveContainer" containerID="24802c4b138f2b463b83fc6319f05e8177ca461cb095f40f9343f93ef378523c" Oct 30 00:41:19.597108 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.596973 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-djplp" Oct 30 00:41:19.643044 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.642971 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" podStartSLOduration=1.6429524309999999 podStartE2EDuration="1.642952431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:19.638926289 +0000 UTC m=+418.210852551" watchObservedRunningTime="2026-10-30 00:41:19.642952431 +0000 UTC m=+418.214878654" Oct 30 00:41:19.686223 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:19.686159 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" podStartSLOduration=1.686143111 podStartE2EDuration="1.686143111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:19.684512174 +0000 UTC m=+418.256438395" watchObservedRunningTime="2026-10-30 00:41:19.686143111 +0000 UTC m=+418.258069332" Oct 30 00:41:30.606173 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:30.606140 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:30.606552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:30.606249 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-wjgtm" Oct 30 00:41:32.996172 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:32.996132 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8"] Oct 30 00:41:32.996595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:32.996342 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" podUID="f89cd976-4ca1-4316-b316-8b5d2c13b6ca" containerName="manager" containerID="cri-o://835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef" gracePeriod=10 Oct 30 00:41:33.259563 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.259494 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:33.392800 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.392763 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-kube-api-access-n5b5q\" (UniqueName: \"kubernetes.io/projected/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-kube-api-access-n5b5q\") pod \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\" (UID: \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\") " Oct 30 00:41:33.392953 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.392838 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-extensions-socket-volume\") pod \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\" (UID: \"f89cd976-4ca1-4316-b316-8b5d2c13b6ca\") " Oct 30 00:41:33.393210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.393181 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-extensions-socket-volume" pod "f89cd976-4ca1-4316-b316-8b5d2c13b6ca" (UID: "f89cd976-4ca1-4316-b316-8b5d2c13b6ca"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:41:33.394997 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.394970 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-kube-api-access-n5b5q" pod "f89cd976-4ca1-4316-b316-8b5d2c13b6ca" (UID: "f89cd976-4ca1-4316-b316-8b5d2c13b6ca"). InnerVolumeSpecName "kube-api-access-n5b5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:41:33.493909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.493872 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5b5q\" (UniqueName: \"kubernetes.io/projected/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-kube-api-access-n5b5q\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:41:33.493909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.493900 4927 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f89cd976-4ca1-4316-b316-8b5d2c13b6ca-extensions-socket-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:41:33.668401 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.668362 4927 generic.go:356] "Generic (PLEG): container finished" podID="f89cd976-4ca1-4316-b316-8b5d2c13b6ca" containerID="835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef" exitCode=0 Oct 30 00:41:33.668401 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.668402 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" event={"ID":"f89cd976-4ca1-4316-b316-8b5d2c13b6ca","Type":"ContainerDied","Data":"835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef"} Oct 30 00:41:33.668650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.668412 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" Oct 30 00:41:33.668650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.668444 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8" event={"ID":"f89cd976-4ca1-4316-b316-8b5d2c13b6ca","Type":"ContainerDied","Data":"8922b6fd71a9036af086c0582bf3e706499f56b1e26c104e0a77de804b385f81"} Oct 30 00:41:33.668650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.668467 4927 scope.go:122] "RemoveContainer" containerID="835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef" Oct 30 00:41:33.677233 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.677208 4927 scope.go:122] "RemoveContainer" containerID="835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef" Oct 30 00:41:33.677526 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:33.677502 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef\": container with ID starting with 835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef not found: ID does not exist" containerID="835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef" Oct 30 00:41:33.677639 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.677540 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef"} err="failed to get container status \"835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef\": rpc error: code = NotFound desc = could not find container \"835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef\": container with ID starting with 835175243c3d5ecbd81dac6098612a088bebfa645b50b29638c3a992e362dcef not found: ID does not exist" Oct 30 00:41:33.722046 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.722007 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8"] Oct 30 00:41:33.733935 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:33.733889 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-n4qh8"] Oct 30 00:41:34.441378 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:41:34.441338 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:41:35.569180 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:35.569144 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89cd976-4ca1-4316-b316-8b5d2c13b6ca" path="/var/lib/kubelet/pods/f89cd976-4ca1-4316-b316-8b5d2c13b6ca/volumes" Oct 30 00:41:50.345429 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.345389 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb"] Oct 30 00:41:50.346050 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.345989 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f89cd976-4ca1-4316-b316-8b5d2c13b6ca" containerName="manager" Oct 30 00:41:50.346050 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.346011 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89cd976-4ca1-4316-b316-8b5d2c13b6ca" containerName="manager" Oct 30 00:41:50.346168 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.346146 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89cd976-4ca1-4316-b316-8b5d2c13b6ca" containerName="manager" Oct 30 00:41:50.349690 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.349664 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.373615 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.373559 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb"] Oct 30 00:41:50.388681 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.388649 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-76qjj\"" Oct 30 00:41:50.443813 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.443775 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.443813 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.443810 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.444049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.443837 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.444049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.443919 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.444049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.443945 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.444049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.444005 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86h2\" (UniqueName: \"kubernetes.io/projected/fd38e17a-6d35-4f06-8ac3-59f578780fe3-kube-api-access-j86h2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.444049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.444036 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.444229 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.444057 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.444229 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.444081 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.544466 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.544429 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86h2\" (UniqueName: \"kubernetes.io/projected/fd38e17a-6d35-4f06-8ac3-59f578780fe3-kube-api-access-j86h2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.544668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.544476 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.544668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.544499 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.544668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.544521 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.544668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.544601 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.544668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.544631 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.544941 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.544668 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.545044 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.544811 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.545215 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.545056 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.545215 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.545078 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.545215 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.545087 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.545215 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.545121 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.545488 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.545306 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.546126 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.546103 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.547024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.547005 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.547113 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.547093 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.566921 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.566889 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd38e17a-6d35-4f06-8ac3-59f578780fe3-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.567087 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.567065 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86h2\" (UniqueName: \"kubernetes.io/projected/fd38e17a-6d35-4f06-8ac3-59f578780fe3-kube-api-access-j86h2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb\" (UID: \"fd38e17a-6d35-4f06-8ac3-59f578780fe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.665504 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.665463 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:50.990310 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.990280 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb"] Oct 30 00:41:50.992290 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:41:50.992259 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd38e17a_6d35_4f06_8ac3_59f578780fe3.slice/crio-7b6935b5dd67484e4db196f42609c5708674e67c02f3b97847ef648d644eb18d WatchSource:0}: Error finding container 7b6935b5dd67484e4db196f42609c5708674e67c02f3b97847ef648d644eb18d: Status 404 returned error can't find the container with id 7b6935b5dd67484e4db196f42609c5708674e67c02f3b97847ef648d644eb18d Oct 30 00:41:50.994505 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.994466 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:41:50.994599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.994544 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:41:50.994599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:50.994585 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:41:51.768932 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:51.768889 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" event={"ID":"fd38e17a-6d35-4f06-8ac3-59f578780fe3","Type":"ContainerStarted","Data":"cc704affeaff0f75671a22e2336331dead1ef1be50ad5a637a5f449abddf325d"} Oct 30 00:41:51.769348 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:51.768952 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" event={"ID":"fd38e17a-6d35-4f06-8ac3-59f578780fe3","Type":"ContainerStarted","Data":"7b6935b5dd67484e4db196f42609c5708674e67c02f3b97847ef648d644eb18d"} Oct 30 00:41:51.814695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:51.814637 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podStartSLOduration=1.814622327 podStartE2EDuration="1.814622327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:51.812051884 +0000 UTC m=+450.383978106" watchObservedRunningTime="2026-10-30 00:41:51.814622327 +0000 UTC m=+450.386548551" Oct 30 00:41:52.666367 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:52.666330 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:52.671641 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:52.671616 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:52.774138 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:52.774102 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:41:52.775652 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:41:52.775626 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:42:04.442095 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:42:04.442051 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:42:05.253855 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.253815 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xtpwm"] Oct 30 00:42:05.259824 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.259791 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:05.275016 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.274983 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 00:42:05.277933 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.277904 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xtpwm"] Oct 30 00:42:05.286051 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.286021 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qsgkj\"" Oct 30 00:42:05.359962 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.359925 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q25c4\" (UniqueName: \"kubernetes.io/projected/81f6b1d9-469d-41bb-bda1-99ee094349ff-kube-api-access-q25c4\") pod \"limitador-limitador-74dfcfb894-xtpwm\" (UID: \"81f6b1d9-469d-41bb-bda1-99ee094349ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:05.360125 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.359969 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f6b1d9-469d-41bb-bda1-99ee094349ff-config-file\") pod \"limitador-limitador-74dfcfb894-xtpwm\" (UID: \"81f6b1d9-469d-41bb-bda1-99ee094349ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:05.390050 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.390015 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xtpwm"] Oct 30 00:42:05.461130 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.461093 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25c4\" (UniqueName: \"kubernetes.io/projected/81f6b1d9-469d-41bb-bda1-99ee094349ff-kube-api-access-q25c4\") pod \"limitador-limitador-74dfcfb894-xtpwm\" (UID: \"81f6b1d9-469d-41bb-bda1-99ee094349ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:05.461130 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.461129 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f6b1d9-469d-41bb-bda1-99ee094349ff-config-file\") pod \"limitador-limitador-74dfcfb894-xtpwm\" (UID: \"81f6b1d9-469d-41bb-bda1-99ee094349ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:05.461766 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.461745 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f6b1d9-469d-41bb-bda1-99ee094349ff-config-file\") pod \"limitador-limitador-74dfcfb894-xtpwm\" (UID: \"81f6b1d9-469d-41bb-bda1-99ee094349ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:05.483367 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.483333 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q25c4\" (UniqueName: \"kubernetes.io/projected/81f6b1d9-469d-41bb-bda1-99ee094349ff-kube-api-access-q25c4\") pod \"limitador-limitador-74dfcfb894-xtpwm\" (UID: \"81f6b1d9-469d-41bb-bda1-99ee094349ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:05.574841 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.574760 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:05.882499 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:05.882309 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xtpwm"] Oct 30 00:42:05.885080 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:42:05.885040 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f6b1d9_469d_41bb_bda1_99ee094349ff.slice/crio-5d1ae190e757a472cc063928fa5b7ccd560a0b75cfb82f23d2282df7349c714d WatchSource:0}: Error finding container 5d1ae190e757a472cc063928fa5b7ccd560a0b75cfb82f23d2282df7349c714d: Status 404 returned error can't find the container with id 5d1ae190e757a472cc063928fa5b7ccd560a0b75cfb82f23d2282df7349c714d Oct 30 00:42:06.845783 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:06.845736 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" event={"ID":"81f6b1d9-469d-41bb-bda1-99ee094349ff","Type":"ContainerStarted","Data":"5d1ae190e757a472cc063928fa5b7ccd560a0b75cfb82f23d2282df7349c714d"} Oct 30 00:42:07.852639 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:07.852599 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" event={"ID":"81f6b1d9-469d-41bb-bda1-99ee094349ff","Type":"ContainerStarted","Data":"8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908"} Oct 30 00:42:07.853080 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:07.852728 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:07.893698 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:07.893636 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" podStartSLOduration=1.586671034 podStartE2EDuration="2.893620781s" totalImagesPullingTime="1.306949747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:07.891365948 +0000 UTC m=+466.463292172" watchObservedRunningTime="2026-10-30 00:42:07.893620781 +0000 UTC m=+466.465547002" Oct 30 00:42:18.859165 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:18.859130 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:20.180237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.180198 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xtpwm"] Oct 30 00:42:20.180675 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.180455 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" podUID="81f6b1d9-469d-41bb-bda1-99ee094349ff" containerName="limitador" containerID="cri-o://8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908" gracePeriod=30 Oct 30 00:42:20.726770 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.726741 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:20.799525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.799421 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f6b1d9-469d-41bb-bda1-99ee094349ff-kube-api-access-q25c4\" (UniqueName: \"kubernetes.io/projected/81f6b1d9-469d-41bb-bda1-99ee094349ff-kube-api-access-q25c4\") pod \"81f6b1d9-469d-41bb-bda1-99ee094349ff\" (UID: \"81f6b1d9-469d-41bb-bda1-99ee094349ff\") " Oct 30 00:42:20.799525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.799480 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f6b1d9-469d-41bb-bda1-99ee094349ff-config-file\" (UniqueName: \"kubernetes.io/configmap/81f6b1d9-469d-41bb-bda1-99ee094349ff-config-file\") pod \"81f6b1d9-469d-41bb-bda1-99ee094349ff\" (UID: \"81f6b1d9-469d-41bb-bda1-99ee094349ff\") " Oct 30 00:42:20.799936 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.799909 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f6b1d9-469d-41bb-bda1-99ee094349ff-config-file" pod "81f6b1d9-469d-41bb-bda1-99ee094349ff" (UID: "81f6b1d9-469d-41bb-bda1-99ee094349ff"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:42:20.801766 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.801743 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f6b1d9-469d-41bb-bda1-99ee094349ff-kube-api-access-q25c4" pod "81f6b1d9-469d-41bb-bda1-99ee094349ff" (UID: "81f6b1d9-469d-41bb-bda1-99ee094349ff"). InnerVolumeSpecName "kube-api-access-q25c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:42:20.900845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.900803 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q25c4\" (UniqueName: \"kubernetes.io/projected/81f6b1d9-469d-41bb-bda1-99ee094349ff-kube-api-access-q25c4\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:42:20.900845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.900837 4927 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f6b1d9-469d-41bb-bda1-99ee094349ff-config-file\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:42:20.921508 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.921471 4927 generic.go:356] "Generic (PLEG): container finished" podID="81f6b1d9-469d-41bb-bda1-99ee094349ff" containerID="8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908" exitCode=0 Oct 30 00:42:20.921787 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.921529 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" Oct 30 00:42:20.921787 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.921559 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" event={"ID":"81f6b1d9-469d-41bb-bda1-99ee094349ff","Type":"ContainerDied","Data":"8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908"} Oct 30 00:42:20.921787 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.921637 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xtpwm" event={"ID":"81f6b1d9-469d-41bb-bda1-99ee094349ff","Type":"ContainerDied","Data":"5d1ae190e757a472cc063928fa5b7ccd560a0b75cfb82f23d2282df7349c714d"} Oct 30 00:42:20.921787 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.921659 4927 scope.go:122] "RemoveContainer" containerID="8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908" Oct 30 00:42:20.930829 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.930812 4927 scope.go:122] "RemoveContainer" containerID="8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908" Oct 30 00:42:20.931141 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:42:20.931120 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908\": container with ID starting with 8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908 not found: ID does not exist" containerID="8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908" Oct 30 00:42:20.931205 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.931156 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908"} err="failed to get container status \"8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908\": rpc error: code = NotFound desc = could not find container \"8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908\": container with ID starting with 8fa2789387c9a24ae12b734129d59f4745673d941fc8e245eede6a73c79fa908 not found: ID does not exist" Oct 30 00:42:20.970060 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.970022 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xtpwm"] Oct 30 00:42:20.974223 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:20.974195 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xtpwm"] Oct 30 00:42:21.569440 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:21.569409 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f6b1d9-469d-41bb-bda1-99ee094349ff" path="/var/lib/kubelet/pods/81f6b1d9-469d-41bb-bda1-99ee094349ff/volumes" Oct 30 00:42:26.400843 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.400801 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx"] Oct 30 00:42:26.401255 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.401183 4927 cpu_manager.go:414] "containerMap: removing container" podUID="81f6b1d9-469d-41bb-bda1-99ee094349ff" containerName="limitador" Oct 30 00:42:26.401255 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.401194 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f6b1d9-469d-41bb-bda1-99ee094349ff" containerName="limitador" Oct 30 00:42:26.401327 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.401287 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f6b1d9-469d-41bb-bda1-99ee094349ff" containerName="limitador" Oct 30 00:42:26.405703 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.405683 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:26.425490 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.425457 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx"] Oct 30 00:42:26.433618 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.433582 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8vl6v\"" Oct 30 00:42:26.433618 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.433593 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:42:26.433858 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.433662 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:42:26.459679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.459644 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 00:42:26.552867 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.552832 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e45d6d94-9a44-42c8-a646-210e06552ffa-data\") pod \"postgres-7cdbf6dc44-bcxfx\" (UID: \"e45d6d94-9a44-42c8-a646-210e06552ffa\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:26.553073 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.552899 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djcrt\" (UniqueName: \"kubernetes.io/projected/e45d6d94-9a44-42c8-a646-210e06552ffa-kube-api-access-djcrt\") pod \"postgres-7cdbf6dc44-bcxfx\" (UID: \"e45d6d94-9a44-42c8-a646-210e06552ffa\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:26.654448 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.654348 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djcrt\" (UniqueName: \"kubernetes.io/projected/e45d6d94-9a44-42c8-a646-210e06552ffa-kube-api-access-djcrt\") pod \"postgres-7cdbf6dc44-bcxfx\" (UID: \"e45d6d94-9a44-42c8-a646-210e06552ffa\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:26.654650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.654454 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e45d6d94-9a44-42c8-a646-210e06552ffa-data\") pod \"postgres-7cdbf6dc44-bcxfx\" (UID: \"e45d6d94-9a44-42c8-a646-210e06552ffa\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:26.654951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.654930 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e45d6d94-9a44-42c8-a646-210e06552ffa-data\") pod \"postgres-7cdbf6dc44-bcxfx\" (UID: \"e45d6d94-9a44-42c8-a646-210e06552ffa\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:26.681964 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.681927 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djcrt\" (UniqueName: \"kubernetes.io/projected/e45d6d94-9a44-42c8-a646-210e06552ffa-kube-api-access-djcrt\") pod \"postgres-7cdbf6dc44-bcxfx\" (UID: \"e45d6d94-9a44-42c8-a646-210e06552ffa\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:26.718433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:26.718397 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:27.027770 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:27.027743 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx"] Oct 30 00:42:27.030002 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:42:27.029975 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45d6d94_9a44_42c8_a646_210e06552ffa.slice/crio-d2167e263f5f6f074555f975e48391b842c8de03723d2aa2b551e28b8790026d WatchSource:0}: Error finding container d2167e263f5f6f074555f975e48391b842c8de03723d2aa2b551e28b8790026d: Status 404 returned error can't find the container with id d2167e263f5f6f074555f975e48391b842c8de03723d2aa2b551e28b8790026d Oct 30 00:42:27.960976 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:27.960931 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" event={"ID":"e45d6d94-9a44-42c8-a646-210e06552ffa","Type":"ContainerStarted","Data":"d2167e263f5f6f074555f975e48391b842c8de03723d2aa2b551e28b8790026d"} Oct 30 00:42:32.991368 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:32.991266 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" event={"ID":"e45d6d94-9a44-42c8-a646-210e06552ffa","Type":"ContainerStarted","Data":"3849a4106687b8026299b0d3956a289e3bb3a5e0e24b6997434f4232404c1348"} Oct 30 00:42:32.991860 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:32.991414 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:33.030590 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:33.030528 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" podStartSLOduration=2.121213894 podStartE2EDuration="7.030513545s" totalImagesPullingTime="4.909299651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:33.028980791 +0000 UTC m=+491.600907026" watchObservedRunningTime="2026-10-30 00:42:33.030513545 +0000 UTC m=+491.602439796" Oct 30 00:42:34.442217 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:42:34.442177 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:42:39.028236 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:39.028202 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bcxfx" Oct 30 00:42:40.002156 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.002115 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-m5jgs"] Oct 30 00:42:40.006922 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.006901 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:40.027166 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.027131 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-m5jgs"] Oct 30 00:42:40.032446 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.032420 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8h6tt\"" Oct 30 00:42:40.032970 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.032469 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 00:42:40.129539 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.129500 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-m5jgs"] Oct 30 00:42:40.129759 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:42:40.129739 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fvj95 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" podUID="0b161b15-5c2c-417b-81fe-b51a2cf09da2" Oct 30 00:42:40.184905 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.184870 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b161b15-5c2c-417b-81fe-b51a2cf09da2-tls-cert\") pod \"authorino-6889cbd8c4-m5jgs\" (UID: \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\") " pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:40.185073 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.184912 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvj95\" (UniqueName: \"kubernetes.io/projected/0b161b15-5c2c-417b-81fe-b51a2cf09da2-kube-api-access-fvj95\") pod \"authorino-6889cbd8c4-m5jgs\" (UID: \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\") " pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:40.286610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.286475 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b161b15-5c2c-417b-81fe-b51a2cf09da2-tls-cert\") pod \"authorino-6889cbd8c4-m5jgs\" (UID: \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\") " pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:40.286610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.286523 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvj95\" (UniqueName: \"kubernetes.io/projected/0b161b15-5c2c-417b-81fe-b51a2cf09da2-kube-api-access-fvj95\") pod \"authorino-6889cbd8c4-m5jgs\" (UID: \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\") " pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:40.289241 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.289216 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b161b15-5c2c-417b-81fe-b51a2cf09da2-tls-cert\") pod \"authorino-6889cbd8c4-m5jgs\" (UID: \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\") " pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:40.309425 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:40.309389 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvj95\" (UniqueName: \"kubernetes.io/projected/0b161b15-5c2c-417b-81fe-b51a2cf09da2-kube-api-access-fvj95\") pod \"authorino-6889cbd8c4-m5jgs\" (UID: \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\") " pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:41.033050 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:41.033017 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:41.041332 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:41.041306 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:41.195252 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:41.195218 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b161b15-5c2c-417b-81fe-b51a2cf09da2-kube-api-access-fvj95\" (UniqueName: \"kubernetes.io/projected/0b161b15-5c2c-417b-81fe-b51a2cf09da2-kube-api-access-fvj95\") pod \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\" (UID: \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\") " Oct 30 00:42:41.195441 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:41.195379 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b161b15-5c2c-417b-81fe-b51a2cf09da2-tls-cert\" (UniqueName: \"kubernetes.io/secret/0b161b15-5c2c-417b-81fe-b51a2cf09da2-tls-cert\") pod \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\" (UID: \"0b161b15-5c2c-417b-81fe-b51a2cf09da2\") " Oct 30 00:42:41.197614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:41.197585 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b161b15-5c2c-417b-81fe-b51a2cf09da2-tls-cert" pod "0b161b15-5c2c-417b-81fe-b51a2cf09da2" (UID: "0b161b15-5c2c-417b-81fe-b51a2cf09da2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:42:41.197737 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:41.197671 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b161b15-5c2c-417b-81fe-b51a2cf09da2-kube-api-access-fvj95" pod "0b161b15-5c2c-417b-81fe-b51a2cf09da2" (UID: "0b161b15-5c2c-417b-81fe-b51a2cf09da2"). InnerVolumeSpecName "kube-api-access-fvj95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:42:41.296360 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:41.296268 4927 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b161b15-5c2c-417b-81fe-b51a2cf09da2-tls-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:42:41.296360 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:41.296301 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvj95\" (UniqueName: \"kubernetes.io/projected/0b161b15-5c2c-417b-81fe-b51a2cf09da2-kube-api-access-fvj95\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:42:42.038976 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:42.038942 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-m5jgs" Oct 30 00:42:42.120283 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:42.120236 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-m5jgs"] Oct 30 00:42:42.124327 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:42.124295 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-m5jgs"] Oct 30 00:42:43.569800 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:43.569768 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b161b15-5c2c-417b-81fe-b51a2cf09da2" path="/var/lib/kubelet/pods/0b161b15-5c2c-417b-81fe-b51a2cf09da2/volumes" Oct 30 00:42:48.612649 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.612615 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-8ht5j"] Oct 30 00:42:48.616944 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.616920 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.631274 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.631249 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5nfwr\"" Oct 30 00:42:48.631944 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.631922 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 00:42:48.642899 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.642871 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 00:42:48.642899 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.642885 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 00:42:48.645542 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.645515 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-8ht5j"] Oct 30 00:42:48.661995 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.661965 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6qr\" (UniqueName: \"kubernetes.io/projected/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-kube-api-access-bc6qr\") pod \"maas-controller-577f596567-8ht5j\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.662142 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.662000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-metrics-certs\") pod \"maas-controller-577f596567-8ht5j\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.662142 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.662041 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-webhook-cert\") pod \"maas-controller-577f596567-8ht5j\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.762639 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.762603 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6qr\" (UniqueName: \"kubernetes.io/projected/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-kube-api-access-bc6qr\") pod \"maas-controller-577f596567-8ht5j\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.762814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.762651 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-metrics-certs\") pod \"maas-controller-577f596567-8ht5j\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.762814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.762697 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-webhook-cert\") pod \"maas-controller-577f596567-8ht5j\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.765293 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.765266 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-metrics-certs\") pod \"maas-controller-577f596567-8ht5j\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.765419 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.765295 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-webhook-cert\") pod \"maas-controller-577f596567-8ht5j\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.785911 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.785881 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6qr\" (UniqueName: \"kubernetes.io/projected/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-kube-api-access-bc6qr\") pod \"maas-controller-577f596567-8ht5j\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:48.930235 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:48.930176 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:49.001508 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.001460 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f6db8855d-hlvbv"] Oct 30 00:42:49.006674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.006647 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.024973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.024943 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f6db8855d-hlvbv"] Oct 30 00:42:49.064664 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.064627 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-metrics-certs\") pod \"maas-controller-f6db8855d-hlvbv\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.064843 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.064691 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprk9\" (UniqueName: \"kubernetes.io/projected/429f70ee-9170-474d-92be-aef90339d241-kube-api-access-nprk9\") pod \"maas-controller-f6db8855d-hlvbv\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.064843 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.064747 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-webhook-cert\") pod \"maas-controller-f6db8855d-hlvbv\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.165276 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.165237 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-webhook-cert\") pod \"maas-controller-f6db8855d-hlvbv\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.165459 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.165334 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-metrics-certs\") pod \"maas-controller-f6db8855d-hlvbv\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.165459 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.165383 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprk9\" (UniqueName: \"kubernetes.io/projected/429f70ee-9170-474d-92be-aef90339d241-kube-api-access-nprk9\") pod \"maas-controller-f6db8855d-hlvbv\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.167881 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.167852 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-metrics-certs\") pod \"maas-controller-f6db8855d-hlvbv\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.167881 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.167868 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-webhook-cert\") pod \"maas-controller-f6db8855d-hlvbv\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.188307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.188238 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprk9\" (UniqueName: \"kubernetes.io/projected/429f70ee-9170-474d-92be-aef90339d241-kube-api-access-nprk9\") pod \"maas-controller-f6db8855d-hlvbv\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.232472 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.232444 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-8ht5j"] Oct 30 00:42:49.235401 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:42:49.235371 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281d9e07_0bba_4fd9_bac9_74f2cbeef21d.slice/crio-ed86c6abd62e9a66991f4bea86aa706fb8bc45039319a37b76188df3699d8494 WatchSource:0}: Error finding container ed86c6abd62e9a66991f4bea86aa706fb8bc45039319a37b76188df3699d8494: Status 404 returned error can't find the container with id ed86c6abd62e9a66991f4bea86aa706fb8bc45039319a37b76188df3699d8494 Oct 30 00:42:49.326142 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.326098 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:49.642351 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:49.642325 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f6db8855d-hlvbv"] Oct 30 00:42:49.645106 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:42:49.645078 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429f70ee_9170_474d_92be_aef90339d241.slice/crio-fe0050ae31d3aa2373f1447277bf662cbd716a0a69771f493a1d563113568caa WatchSource:0}: Error finding container fe0050ae31d3aa2373f1447277bf662cbd716a0a69771f493a1d563113568caa: Status 404 returned error can't find the container with id fe0050ae31d3aa2373f1447277bf662cbd716a0a69771f493a1d563113568caa Oct 30 00:42:50.086642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:50.086461 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-8ht5j" event={"ID":"281d9e07-0bba-4fd9-bac9-74f2cbeef21d","Type":"ContainerStarted","Data":"ed86c6abd62e9a66991f4bea86aa706fb8bc45039319a37b76188df3699d8494"} Oct 30 00:42:50.088532 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:50.088456 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-hlvbv" event={"ID":"429f70ee-9170-474d-92be-aef90339d241","Type":"ContainerStarted","Data":"fe0050ae31d3aa2373f1447277bf662cbd716a0a69771f493a1d563113568caa"} Oct 30 00:42:52.102249 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:52.102206 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-8ht5j" event={"ID":"281d9e07-0bba-4fd9-bac9-74f2cbeef21d","Type":"ContainerStarted","Data":"9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a"} Oct 30 00:42:52.102752 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:52.102360 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:42:52.146842 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:52.146781 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-8ht5j" podStartSLOduration=1.4800283410000001 podStartE2EDuration="4.14676156s" totalImagesPullingTime="2.666733219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:52.145961092 +0000 UTC m=+510.717887315" watchObservedRunningTime="2026-10-30 00:42:52.14676156 +0000 UTC m=+510.718687784" Oct 30 00:42:53.108513 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:53.108476 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-hlvbv" event={"ID":"429f70ee-9170-474d-92be-aef90339d241","Type":"ContainerStarted","Data":"c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c"} Oct 30 00:42:53.109106 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:53.108556 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:42:53.152132 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:53.152073 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f6db8855d-hlvbv" podStartSLOduration=2.53062278 podStartE2EDuration="5.152056692s" totalImagesPullingTime="2.621433912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:53.150309897 +0000 UTC m=+511.722236120" watchObservedRunningTime="2026-10-30 00:42:53.152056692 +0000 UTC m=+511.723982954" Oct 30 00:42:55.932247 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:55.932205 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc"] Oct 30 00:42:55.936490 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:55.936465 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:55.950534 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:55.950502 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 00:42:55.961524 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:55.961496 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 00:42:55.961728 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:55.961683 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 00:42:55.961728 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:55.961721 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-gqh4p\"" Oct 30 00:42:55.964182 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:55.964156 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc"] Oct 30 00:42:56.032600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.032536 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1a5359b7-b845-4451-a24c-b62c4dac56df-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.032789 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.032615 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5359b7-b845-4451-a24c-b62c4dac56df-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.032789 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.032686 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwqz\" (UniqueName: \"kubernetes.io/projected/1a5359b7-b845-4451-a24c-b62c4dac56df-kube-api-access-qtwqz\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.032789 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.032729 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a5359b7-b845-4451-a24c-b62c4dac56df-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.133925 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.133887 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5359b7-b845-4451-a24c-b62c4dac56df-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.133925 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.133931 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwqz\" (UniqueName: \"kubernetes.io/projected/1a5359b7-b845-4451-a24c-b62c4dac56df-kube-api-access-qtwqz\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.134206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.134057 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a5359b7-b845-4451-a24c-b62c4dac56df-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.134267 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.134206 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1a5359b7-b845-4451-a24c-b62c4dac56df-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.134769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.134742 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5359b7-b845-4451-a24c-b62c4dac56df-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.135063 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.135044 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1a5359b7-b845-4451-a24c-b62c4dac56df-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.136838 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.136821 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a5359b7-b845-4451-a24c-b62c4dac56df-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.156355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.156324 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwqz\" (UniqueName: \"kubernetes.io/projected/1a5359b7-b845-4451-a24c-b62c4dac56df-kube-api-access-qtwqz\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-wx4nc\" (UID: \"1a5359b7-b845-4451-a24c-b62c4dac56df\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.249730 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.249634 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:42:56.554491 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:56.554462 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc"] Oct 30 00:42:56.556387 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:42:56.556350 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5359b7_b845_4451_a24c_b62c4dac56df.slice/crio-ae7622910230576cfd2b1c5d6d92a3abd8c46b7e9a1580fee9819436b1994d1c WatchSource:0}: Error finding container ae7622910230576cfd2b1c5d6d92a3abd8c46b7e9a1580fee9819436b1994d1c: Status 404 returned error can't find the container with id ae7622910230576cfd2b1c5d6d92a3abd8c46b7e9a1580fee9819436b1994d1c Oct 30 00:42:57.131016 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:42:57.130979 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" event={"ID":"1a5359b7-b845-4451-a24c-b62c4dac56df","Type":"ContainerStarted","Data":"ae7622910230576cfd2b1c5d6d92a3abd8c46b7e9a1580fee9819436b1994d1c"} Oct 30 00:43:03.112093 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:03.112061 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:43:04.117165 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:04.117131 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 00:43:04.187532 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:04.187483 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-8ht5j"] Oct 30 00:43:04.188034 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:04.187906 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-8ht5j" podUID="281d9e07-0bba-4fd9-bac9-74f2cbeef21d" containerName="manager" containerID="cri-o://9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a" gracePeriod=30 Oct 30 00:43:04.445928 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:43:04.445884 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:43:04.984115 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:04.984087 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:43:05.122745 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.122659 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-metrics-certs\") pod \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " Oct 30 00:43:05.122745 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.122716 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-kube-api-access-bc6qr\" (UniqueName: \"kubernetes.io/projected/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-kube-api-access-bc6qr\") pod \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " Oct 30 00:43:05.123299 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.122852 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-webhook-cert\") pod \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\" (UID: \"281d9e07-0bba-4fd9-bac9-74f2cbeef21d\") " Oct 30 00:43:05.125902 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.125851 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-kube-api-access-bc6qr" pod "281d9e07-0bba-4fd9-bac9-74f2cbeef21d" (UID: "281d9e07-0bba-4fd9-bac9-74f2cbeef21d"). InnerVolumeSpecName "kube-api-access-bc6qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:43:05.130601 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.126184 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-metrics-certs" pod "281d9e07-0bba-4fd9-bac9-74f2cbeef21d" (UID: "281d9e07-0bba-4fd9-bac9-74f2cbeef21d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:05.130601 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.126604 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-webhook-cert" pod "281d9e07-0bba-4fd9-bac9-74f2cbeef21d" (UID: "281d9e07-0bba-4fd9-bac9-74f2cbeef21d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:05.185512 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.185472 4927 generic.go:356] "Generic (PLEG): container finished" podID="281d9e07-0bba-4fd9-bac9-74f2cbeef21d" containerID="9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a" exitCode=0 Oct 30 00:43:05.185712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.185631 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-8ht5j" event={"ID":"281d9e07-0bba-4fd9-bac9-74f2cbeef21d","Type":"ContainerDied","Data":"9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a"} Oct 30 00:43:05.185712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.185685 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-8ht5j" event={"ID":"281d9e07-0bba-4fd9-bac9-74f2cbeef21d","Type":"ContainerDied","Data":"ed86c6abd62e9a66991f4bea86aa706fb8bc45039319a37b76188df3699d8494"} Oct 30 00:43:05.185840 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.185715 4927 scope.go:122] "RemoveContainer" containerID="9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a" Oct 30 00:43:05.186086 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.186067 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-8ht5j" Oct 30 00:43:05.198945 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.198819 4927 scope.go:122] "RemoveContainer" containerID="9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a" Oct 30 00:43:05.199286 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:43:05.199259 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a\": container with ID starting with 9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a not found: ID does not exist" containerID="9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a" Oct 30 00:43:05.199389 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.199296 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a"} err="failed to get container status \"9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a\": rpc error: code = NotFound desc = could not find container \"9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a\": container with ID starting with 9be15bd9325d7456e276c5ae88141c2f829a7cc452fdbc4ed3c709374a8aec6a not found: ID does not exist" Oct 30 00:43:05.223814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.223778 4927 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-webhook-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:43:05.223814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.223812 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:43:05.223814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.223826 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc6qr\" (UniqueName: \"kubernetes.io/projected/281d9e07-0bba-4fd9-bac9-74f2cbeef21d-kube-api-access-bc6qr\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:43:05.236026 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.235986 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-8ht5j"] Oct 30 00:43:05.239361 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.239333 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-8ht5j"] Oct 30 00:43:05.569218 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:05.569187 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="281d9e07-0bba-4fd9-bac9-74f2cbeef21d" path="/var/lib/kubelet/pods/281d9e07-0bba-4fd9-bac9-74f2cbeef21d/volumes" Oct 30 00:43:07.199985 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:07.199945 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" event={"ID":"1a5359b7-b845-4451-a24c-b62c4dac56df","Type":"ContainerStarted","Data":"95cd38ce195bffefa17150920f879c3cf0f3d1fdfb57731dacbf8c165e84aef2"} Oct 30 00:43:07.200397 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:07.200101 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:43:07.242641 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:07.242583 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" podStartSLOduration=1.744655531 podStartE2EDuration="12.242551162s" totalImagesPullingTime="10.497895631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:07.240247676 +0000 UTC m=+525.812173899" watchObservedRunningTime="2026-10-30 00:43:07.242551162 +0000 UTC m=+525.814477383" Oct 30 00:43:18.211731 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:18.211692 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-wx4nc" Oct 30 00:43:26.360105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.360069 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf"] Oct 30 00:43:26.360617 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.360477 4927 cpu_manager.go:414] "containerMap: removing container" podUID="281d9e07-0bba-4fd9-bac9-74f2cbeef21d" containerName="manager" Oct 30 00:43:26.360617 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.360489 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281d9e07-0bba-4fd9-bac9-74f2cbeef21d" containerName="manager" Oct 30 00:43:26.360617 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.360561 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="281d9e07-0bba-4fd9-bac9-74f2cbeef21d" containerName="manager" Oct 30 00:43:26.367297 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.367274 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.393561 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.393524 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 00:43:26.393561 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.393524 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k5bc5\"" Oct 30 00:43:26.393786 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.393607 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 00:43:26.396193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.396165 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf"] Oct 30 00:43:26.524417 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.524373 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-maas-api-tls\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.524417 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.524418 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-metrics-certs\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.524665 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.524609 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77df2802-ed6e-40ec-9dca-32a123a63a71-openshift-service-ca\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.524665 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.524653 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4z5x\" (UniqueName: \"kubernetes.io/projected/77df2802-ed6e-40ec-9dca-32a123a63a71-kube-api-access-c4z5x\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.625585 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.625532 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77df2802-ed6e-40ec-9dca-32a123a63a71-openshift-service-ca\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.625784 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.625615 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4z5x\" (UniqueName: \"kubernetes.io/projected/77df2802-ed6e-40ec-9dca-32a123a63a71-kube-api-access-c4z5x\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.625784 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.625644 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-maas-api-tls\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.625784 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.625659 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-metrics-certs\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.626245 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.626212 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77df2802-ed6e-40ec-9dca-32a123a63a71-openshift-service-ca\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.628205 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.628178 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-metrics-certs\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.628299 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.628206 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-maas-api-tls\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.648459 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.648429 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4z5x\" (UniqueName: \"kubernetes.io/projected/77df2802-ed6e-40ec-9dca-32a123a63a71-kube-api-access-c4z5x\") pod \"maas-api-5c48877c75-m5jgf\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.680224 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.680187 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:26.988868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:26.988837 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf"] Oct 30 00:43:26.990895 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:43:26.990869 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77df2802_ed6e_40ec_9dca_32a123a63a71.slice/crio-f4a00f5a6dd10f61a466c49c5a357ebc14853bd75fa56087eaf6fc243ba5770c WatchSource:0}: Error finding container f4a00f5a6dd10f61a466c49c5a357ebc14853bd75fa56087eaf6fc243ba5770c: Status 404 returned error can't find the container with id f4a00f5a6dd10f61a466c49c5a357ebc14853bd75fa56087eaf6fc243ba5770c Oct 30 00:43:27.304685 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:27.304596 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" event={"ID":"77df2802-ed6e-40ec-9dca-32a123a63a71","Type":"ContainerStarted","Data":"f4a00f5a6dd10f61a466c49c5a357ebc14853bd75fa56087eaf6fc243ba5770c"} Oct 30 00:43:29.318354 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:29.318313 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" event={"ID":"77df2802-ed6e-40ec-9dca-32a123a63a71","Type":"ContainerStarted","Data":"2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f"} Oct 30 00:43:29.318817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:29.318413 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:29.359209 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:29.359147 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" podStartSLOduration=1.9163214179999999 podStartE2EDuration="3.359131767s" totalImagesPullingTime="1.442810349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:29.357630614 +0000 UTC m=+547.929556838" watchObservedRunningTime="2026-10-30 00:43:29.359131767 +0000 UTC m=+547.931057989" Oct 30 00:43:34.443606 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:43:34.443540 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:43:35.331485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:35.331403 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 00:43:38.449862 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:38.429810 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": read tcp 10.132.0.2:37638->10.132.0.45:15021: read: connection reset by peer" start-of-body= Oct 30 00:43:38.449862 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:38.429895 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podUID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": read tcp 10.132.0.2:37638->10.132.0.45:15021: read: connection reset by peer" Oct 30 00:43:39.382640 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:39.382599 4927 generic.go:356] "Generic (PLEG): container finished" podID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerID="cc704affeaff0f75671a22e2336331dead1ef1be50ad5a637a5f449abddf325d" exitCode=0 Oct 30 00:43:39.382825 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:39.382661 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" event={"ID":"fd38e17a-6d35-4f06-8ac3-59f578780fe3","Type":"ContainerDied","Data":"cc704affeaff0f75671a22e2336331dead1ef1be50ad5a637a5f449abddf325d"} Oct 30 00:43:39.383097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:39.383070 4927 scope.go:122] "RemoveContainer" containerID="cc704affeaff0f75671a22e2336331dead1ef1be50ad5a637a5f449abddf325d" Oct 30 00:43:39.384266 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:39.384237 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:39.384376 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:39.384325 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:39.384436 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:39.384371 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:40.411727 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:40.411691 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" event={"ID":"fd38e17a-6d35-4f06-8ac3-59f578780fe3","Type":"ContainerStarted","Data":"c6a97151a4a5ec1a9ecf522d1a5589bda409413e2b2528abbe614f2e68a3e5ca"} Oct 30 00:43:40.666116 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:40.666021 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:43:40.666304 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:40.666280 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:43:40.666346 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:40.666328 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podUID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 00:43:41.665951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:41.665914 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:43:41.666412 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:41.665971 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podUID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 00:43:42.666316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:42.666276 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:43:42.666788 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:42.666334 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podUID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 00:43:43.665953 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:43.665920 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:43:43.666140 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:43.665971 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podUID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 00:43:44.666023 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:44.665979 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:43:44.666501 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:44.666051 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podUID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 00:43:45.669940 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:45.669905 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:43:45.670392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:45.670215 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:43:45.671020 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:43:45.671001 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" Oct 30 00:44:04.442178 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:44:04.442088 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:44:21.638210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:21.638176 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:44:21.638802 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:21.638272 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:44:29.886886 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.886842 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv"] Oct 30 00:44:29.892335 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.892307 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:29.920551 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.920519 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjh9\" (UniqueName: \"kubernetes.io/projected/6eff56d9-2a99-4b86-9ba0-4bb228804526-kube-api-access-ncjh9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:29.920735 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.920600 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:29.920735 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.920656 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:29.920735 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.920711 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:29.920870 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.920795 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:29.920870 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.920831 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eff56d9-2a99-4b86-9ba0-4bb228804526-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:29.922209 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.922187 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-28sms\"" Oct 30 00:44:29.922321 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.922231 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 00:44:29.922321 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.922314 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 00:44:29.922395 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.922326 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 00:44:29.924402 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:29.924380 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv"] Oct 30 00:44:30.022156 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.022119 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjh9\" (UniqueName: \"kubernetes.io/projected/6eff56d9-2a99-4b86-9ba0-4bb228804526-kube-api-access-ncjh9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.022363 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.022178 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.022363 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.022197 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.022363 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.022220 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.022363 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.022237 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.022363 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.022259 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eff56d9-2a99-4b86-9ba0-4bb228804526-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.022712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.022691 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.022790 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.022768 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.022841 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.022826 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.024669 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.024636 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eff56d9-2a99-4b86-9ba0-4bb228804526-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.024962 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.024942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eff56d9-2a99-4b86-9ba0-4bb228804526-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.045229 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.045194 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjh9\" (UniqueName: \"kubernetes.io/projected/6eff56d9-2a99-4b86-9ba0-4bb228804526-kube-api-access-ncjh9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmfpv\" (UID: \"6eff56d9-2a99-4b86-9ba0-4bb228804526\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.204899 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.204798 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:30.515544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.515514 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv"] Oct 30 00:44:30.517890 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:44:30.517861 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eff56d9_2a99_4b86_9ba0_4bb228804526.slice/crio-21ab23161ec4a97567c2f3c7b37846d4e6db4fbbc5f33611148b6856e7e25d2a WatchSource:0}: Error finding container 21ab23161ec4a97567c2f3c7b37846d4e6db4fbbc5f33611148b6856e7e25d2a: Status 404 returned error can't find the container with id 21ab23161ec4a97567c2f3c7b37846d4e6db4fbbc5f33611148b6856e7e25d2a Oct 30 00:44:30.677079 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:30.677040 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" event={"ID":"6eff56d9-2a99-4b86-9ba0-4bb228804526","Type":"ContainerStarted","Data":"21ab23161ec4a97567c2f3c7b37846d4e6db4fbbc5f33611148b6856e7e25d2a"} Oct 30 00:44:32.691728 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:32.691682 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" event={"ID":"6eff56d9-2a99-4b86-9ba0-4bb228804526","Type":"ContainerStarted","Data":"39ecede9ee2e13e4c1ad0054fdc3f5ba36dc2ed9ced03531acb1858a908e0be5"} Oct 30 00:44:32.734397 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:32.734326 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" podStartSLOduration=2.641924538 podStartE2EDuration="3.734304232s" totalImagesPullingTime="1.092379694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:32.733212641 +0000 UTC m=+611.305138864" watchObservedRunningTime="2026-10-30 00:44:32.734304232 +0000 UTC m=+611.306230454" Oct 30 00:44:33.205134 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:33.205087 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:34.446245 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:44:34.446207 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:44:34.496542 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.496506 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw"] Oct 30 00:44:34.502531 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.502498 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.525838 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.525801 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw"] Oct 30 00:44:34.534004 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.533973 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 00:44:34.563955 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.563918 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.564123 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.563961 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.564123 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.563985 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dl67\" (UniqueName: \"kubernetes.io/projected/857af215-3487-4286-aa40-a9e8b01942a8-kube-api-access-8dl67\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.564194 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.564131 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.564194 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.564168 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857af215-3487-4286-aa40-a9e8b01942a8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.564275 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.564247 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.665024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.664984 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.665024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.665027 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.665269 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.665048 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dl67\" (UniqueName: \"kubernetes.io/projected/857af215-3487-4286-aa40-a9e8b01942a8-kube-api-access-8dl67\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.665269 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.665095 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.665269 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.665135 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857af215-3487-4286-aa40-a9e8b01942a8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.665269 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.665177 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.665496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.665468 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.665558 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.665524 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.665669 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.665622 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.667463 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.667442 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857af215-3487-4286-aa40-a9e8b01942a8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.667899 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.667880 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857af215-3487-4286-aa40-a9e8b01942a8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.691271 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.691234 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dl67\" (UniqueName: \"kubernetes.io/projected/857af215-3487-4286-aa40-a9e8b01942a8-kube-api-access-8dl67\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k6ckw\" (UID: \"857af215-3487-4286-aa40-a9e8b01942a8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:34.818744 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:34.818656 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:35.123730 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:35.123702 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw"] Oct 30 00:44:35.125995 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:44:35.125963 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857af215_3487_4286_aa40_a9e8b01942a8.slice/crio-752249d07f204d245eb9c89f5f6f266b854a542030c1fc5722685d9dfe4cbe30 WatchSource:0}: Error finding container 752249d07f204d245eb9c89f5f6f266b854a542030c1fc5722685d9dfe4cbe30: Status 404 returned error can't find the container with id 752249d07f204d245eb9c89f5f6f266b854a542030c1fc5722685d9dfe4cbe30 Oct 30 00:44:35.710086 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:35.709994 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" event={"ID":"857af215-3487-4286-aa40-a9e8b01942a8","Type":"ContainerStarted","Data":"937d1c78cf34138a8efc6f0a0e2a8d7fa453f1fbb4cf9599fbdcc5d55ca1083b"} Oct 30 00:44:35.710086 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:35.710054 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" event={"ID":"857af215-3487-4286-aa40-a9e8b01942a8","Type":"ContainerStarted","Data":"752249d07f204d245eb9c89f5f6f266b854a542030c1fc5722685d9dfe4cbe30"} Oct 30 00:44:35.752753 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:35.752691 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" podStartSLOduration=1.535889868 podStartE2EDuration="1.752674967s" totalImagesPullingTime="216.785099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:35.750166643 +0000 UTC m=+614.322092866" watchObservedRunningTime="2026-10-30 00:44:35.752674967 +0000 UTC m=+614.324601189" Oct 30 00:44:35.819065 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:35.819027 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:40.205778 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:40.205735 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:40.219386 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:40.219348 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:40.751112 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:40.751083 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmfpv" Oct 30 00:44:43.194907 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.194868 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4"] Oct 30 00:44:43.198897 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.198869 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.215420 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.215394 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 00:44:43.228920 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.228886 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4"] Oct 30 00:44:43.247784 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.247746 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.247986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.247793 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.247986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.247814 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.247986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.247940 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.248219 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.248012 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aedaf3b-4fbd-485d-bf58-319230ad826a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.248219 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.248056 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45k6\" (UniqueName: \"kubernetes.io/projected/1aedaf3b-4fbd-485d-bf58-319230ad826a-kube-api-access-l45k6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.349068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.349027 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.349068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.349074 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.349320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.349103 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.349320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.349146 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.349438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.349359 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aedaf3b-4fbd-485d-bf58-319230ad826a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.349438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.349404 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45k6\" (UniqueName: \"kubernetes.io/projected/1aedaf3b-4fbd-485d-bf58-319230ad826a-kube-api-access-l45k6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.349537 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.349467 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.349627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.349544 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.349683 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.349625 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.351405 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.351383 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aedaf3b-4fbd-485d-bf58-319230ad826a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.351784 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.351766 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aedaf3b-4fbd-485d-bf58-319230ad826a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.372127 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.372088 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45k6\" (UniqueName: \"kubernetes.io/projected/1aedaf3b-4fbd-485d-bf58-319230ad826a-kube-api-access-l45k6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4\" (UID: \"1aedaf3b-4fbd-485d-bf58-319230ad826a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.512318 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.512230 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:43.817666 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:43.817637 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4"] Oct 30 00:44:43.819012 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:44:43.818981 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aedaf3b_4fbd_485d_bf58_319230ad826a.slice/crio-6280f3b2a55c5e400b62634ea06ee6388d0648b1c580f9d95073e86c3bd2e2b5 WatchSource:0}: Error finding container 6280f3b2a55c5e400b62634ea06ee6388d0648b1c580f9d95073e86c3bd2e2b5: Status 404 returned error can't find the container with id 6280f3b2a55c5e400b62634ea06ee6388d0648b1c580f9d95073e86c3bd2e2b5 Oct 30 00:44:44.521936 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.521897 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk"] Oct 30 00:44:44.526871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.526839 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.544187 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.544156 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 00:44:44.557723 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.557691 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk"] Oct 30 00:44:44.560801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.560773 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.560945 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.560804 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19cce81-4a48-4b54-b2ac-29ef57f9395c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.561033 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.561006 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.561069 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.561052 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.561108 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.561101 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.561139 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.561120 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w2hc\" (UniqueName: \"kubernetes.io/projected/c19cce81-4a48-4b54-b2ac-29ef57f9395c-kube-api-access-5w2hc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.662219 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.662179 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.662219 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.662220 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.662462 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.662249 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.662462 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.662285 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w2hc\" (UniqueName: \"kubernetes.io/projected/c19cce81-4a48-4b54-b2ac-29ef57f9395c-kube-api-access-5w2hc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.662462 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.662331 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.662462 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.662347 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19cce81-4a48-4b54-b2ac-29ef57f9395c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.662663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.662641 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.662760 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.662739 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.662870 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.662849 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.664739 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.664709 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19cce81-4a48-4b54-b2ac-29ef57f9395c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.664944 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.664928 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19cce81-4a48-4b54-b2ac-29ef57f9395c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.684969 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.684936 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w2hc\" (UniqueName: \"kubernetes.io/projected/c19cce81-4a48-4b54-b2ac-29ef57f9395c-kube-api-access-5w2hc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk\" (UID: \"c19cce81-4a48-4b54-b2ac-29ef57f9395c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:44.764469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.764426 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" event={"ID":"1aedaf3b-4fbd-485d-bf58-319230ad826a","Type":"ContainerStarted","Data":"a818cca5c6d2b6747208a6ad8cd65340d65f89aafd5b70ca2953721f946a8f52"} Oct 30 00:44:44.764668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.764480 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" event={"ID":"1aedaf3b-4fbd-485d-bf58-319230ad826a","Type":"ContainerStarted","Data":"6280f3b2a55c5e400b62634ea06ee6388d0648b1c580f9d95073e86c3bd2e2b5"} Oct 30 00:44:44.804864 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.804752 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podStartSLOduration=1.635910516 podStartE2EDuration="1.804731165s" totalImagesPullingTime="168.820649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:44.804322117 +0000 UTC m=+623.376248338" watchObservedRunningTime="2026-10-30 00:44:44.804731165 +0000 UTC m=+623.376657419" Oct 30 00:44:44.819453 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.819419 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:44.831969 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.831942 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:44.843342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:44.843305 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:45.149971 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:45.149944 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk"] Oct 30 00:44:45.152020 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:44:45.151986 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19cce81_4a48_4b54_b2ac_29ef57f9395c.slice/crio-e8fbc04a7f8fb89cc7d1fa8c741cb42c76ab2ba53ba932bf2dba09ac41272caa WatchSource:0}: Error finding container e8fbc04a7f8fb89cc7d1fa8c741cb42c76ab2ba53ba932bf2dba09ac41272caa: Status 404 returned error can't find the container with id e8fbc04a7f8fb89cc7d1fa8c741cb42c76ab2ba53ba932bf2dba09ac41272caa Oct 30 00:44:45.513244 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:45.513206 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:45.772119 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:45.772009 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" event={"ID":"c19cce81-4a48-4b54-b2ac-29ef57f9395c","Type":"ContainerStarted","Data":"d6175057ef517289314af7254ebc7e10ef62f1223a58aa1d4c77d9dcaaea4574"} Oct 30 00:44:45.772119 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:45.772076 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" event={"ID":"c19cce81-4a48-4b54-b2ac-29ef57f9395c","Type":"ContainerStarted","Data":"e8fbc04a7f8fb89cc7d1fa8c741cb42c76ab2ba53ba932bf2dba09ac41272caa"} Oct 30 00:44:45.783847 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:45.783814 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k6ckw" Oct 30 00:44:45.812589 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:45.812506 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" podStartSLOduration=1.6317846409999999 podStartE2EDuration="1.81248799s" totalImagesPullingTime="180.703349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:45.811906935 +0000 UTC m=+624.383833156" watchObservedRunningTime="2026-10-30 00:44:45.81248799 +0000 UTC m=+624.384414240" Oct 30 00:44:45.843408 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:45.843363 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:53.512432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:53.512392 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:53.525199 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:53.525161 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:53.828938 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:53.828850 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:44:54.844394 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:54.844350 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:54.857130 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:54.857103 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:44:55.839153 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:44:55.839123 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" Oct 30 00:45:04.441883 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:45:04.441830 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:45:05.644295 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.644249 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm"] Oct 30 00:45:05.648210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.648183 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:05.664072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.664048 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2pcqk\"" Oct 30 00:45:05.678995 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.678958 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm"] Oct 30 00:45:05.691090 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.691049 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 00:45:05.765092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.765049 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrzf\" (UniqueName: \"kubernetes.io/projected/1ae9c7c6-32b3-420d-9cfd-6fc7746096d5-kube-api-access-zdrzf\") pod \"payload-pre-processing-7b98bb678c-8q7vm\" (UID: \"1ae9c7c6-32b3-420d-9cfd-6fc7746096d5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:05.765298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.765124 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae9c7c6-32b3-420d-9cfd-6fc7746096d5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8q7vm\" (UID: \"1ae9c7c6-32b3-420d-9cfd-6fc7746096d5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:05.866033 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.865995 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrzf\" (UniqueName: \"kubernetes.io/projected/1ae9c7c6-32b3-420d-9cfd-6fc7746096d5-kube-api-access-zdrzf\") pod \"payload-pre-processing-7b98bb678c-8q7vm\" (UID: \"1ae9c7c6-32b3-420d-9cfd-6fc7746096d5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:05.866224 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.866047 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae9c7c6-32b3-420d-9cfd-6fc7746096d5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8q7vm\" (UID: \"1ae9c7c6-32b3-420d-9cfd-6fc7746096d5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:05.866786 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.866765 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae9c7c6-32b3-420d-9cfd-6fc7746096d5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8q7vm\" (UID: \"1ae9c7c6-32b3-420d-9cfd-6fc7746096d5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:05.888547 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.888513 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrzf\" (UniqueName: \"kubernetes.io/projected/1ae9c7c6-32b3-420d-9cfd-6fc7746096d5-kube-api-access-zdrzf\") pod \"payload-pre-processing-7b98bb678c-8q7vm\" (UID: \"1ae9c7c6-32b3-420d-9cfd-6fc7746096d5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:05.962225 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:05.962132 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:06.118718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.118676 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92"] Oct 30 00:45:06.122655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.122626 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.159332 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.159290 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92"] Oct 30 00:45:06.266604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.266435 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm"] Oct 30 00:45:06.269080 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.269048 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.269230 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:45:06.269077 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae9c7c6_32b3_420d_9cfd_6fc7746096d5.slice/crio-80c167d148c533853ba52c5b5ad8513bfa7252ca2eb88b7a879c9895c990a6d6 WatchSource:0}: Error finding container 80c167d148c533853ba52c5b5ad8513bfa7252ca2eb88b7a879c9895c990a6d6: Status 404 returned error can't find the container with id 80c167d148c533853ba52c5b5ad8513bfa7252ca2eb88b7a879c9895c990a6d6 Oct 30 00:45:06.269230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.269112 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.269230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.269140 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.269350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.269254 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6441bb1-1412-4588-8f9a-200f1d0fee84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.269350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.269277 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.269350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.269307 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.269350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.269329 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.269473 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.269351 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.269473 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.269397 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrmr\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-kube-api-access-nqrmr\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370268 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370229 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrmr\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-kube-api-access-nqrmr\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370268 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370275 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370323 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370357 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370409 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6441bb1-1412-4588-8f9a-200f1d0fee84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370437 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370489 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370524 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370564 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370823 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.370951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370871 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.371088 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.370977 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.371088 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.371002 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.371222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.371204 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6441bb1-1412-4588-8f9a-200f1d0fee84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.373005 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.372980 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.373197 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.373178 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.393004 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.392963 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.393004 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.392995 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrmr\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-kube-api-access-nqrmr\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mkm92\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.437823 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.437785 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:06.760964 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.760936 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92"] Oct 30 00:45:06.763299 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:45:06.763270 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6441bb1_1412_4588_8f9a_200f1d0fee84.slice/crio-cb7f945659f61969be08264f4140f39c5229116853b9bbe2ce4baed79dea3c2a WatchSource:0}: Error finding container cb7f945659f61969be08264f4140f39c5229116853b9bbe2ce4baed79dea3c2a: Status 404 returned error can't find the container with id cb7f945659f61969be08264f4140f39c5229116853b9bbe2ce4baed79dea3c2a Oct 30 00:45:06.765449 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.765419 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:06.765522 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.765487 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:06.765522 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.765517 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:06.890886 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.890842 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" event={"ID":"1ae9c7c6-32b3-420d-9cfd-6fc7746096d5","Type":"ContainerStarted","Data":"80c167d148c533853ba52c5b5ad8513bfa7252ca2eb88b7a879c9895c990a6d6"} Oct 30 00:45:06.894600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.893558 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" event={"ID":"b6441bb1-1412-4588-8f9a-200f1d0fee84","Type":"ContainerStarted","Data":"736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029"} Oct 30 00:45:06.894600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.893656 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" event={"ID":"b6441bb1-1412-4588-8f9a-200f1d0fee84","Type":"ContainerStarted","Data":"cb7f945659f61969be08264f4140f39c5229116853b9bbe2ce4baed79dea3c2a"} Oct 30 00:45:06.940478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:06.939694 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podStartSLOduration=0.939672822 podStartE2EDuration="939.672822ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:06.937879947 +0000 UTC m=+645.509806196" watchObservedRunningTime="2026-10-30 00:45:06.939672822 +0000 UTC m=+645.511599045" Oct 30 00:45:07.438908 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:07.438872 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:07.440704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:07.440674 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:45:07.440852 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:07.440735 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:45:08.439049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:08.439011 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:45:08.439558 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:08.439076 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:45:09.438418 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:09.438382 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:45:09.438662 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:09.438438 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:45:09.921106 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:09.921066 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" event={"ID":"1ae9c7c6-32b3-420d-9cfd-6fc7746096d5","Type":"ContainerStarted","Data":"debf2e8dcf0973026c1b587b8e6ae86cf51843628bd877f01170051db588f7ab"} Oct 30 00:45:09.921522 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:09.921139 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:09.962585 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:09.962495 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" podStartSLOduration=2.3892515149999998 podStartE2EDuration="4.96247991s" totalImagesPullingTime="2.573228395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:09.960115764 +0000 UTC m=+648.532041985" watchObservedRunningTime="2026-10-30 00:45:09.96247991 +0000 UTC m=+648.534406132" Oct 30 00:45:10.438393 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:10.438348 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:45:10.438649 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:10.438416 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:45:11.438593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:11.438528 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:45:11.439004 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:11.438619 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:45:12.439003 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:12.438962 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:45:12.439457 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:12.439020 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:45:13.438147 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:13.438113 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:45:13.438345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:13.438176 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 00:45:15.208174 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:15.208135 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:15.208645 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:15.208519 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:15.210224 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:15.210196 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 00:45:20.930889 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:45:20.930857 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8q7vm" Oct 30 00:45:34.444922 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:45:34.444820 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:46:04.443270 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:46:04.443233 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:46:08.863339 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:08.863292 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c"] Oct 30 00:46:08.867099 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:08.867075 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:08.882173 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:08.882147 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-40ac41\"" Oct 30 00:46:08.892194 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:08.892164 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-40ac41-dockercfg-4zn62\"" Oct 30 00:46:08.894905 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:08.894871 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c"] Oct 30 00:46:08.982482 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:08.982443 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq"] Oct 30 00:46:08.986503 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:08.986470 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:09.014395 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.014133 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd70c464-1507-414d-96d9-a138e4143281-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c\" (UID: \"cd70c464-1507-414d-96d9-a138e4143281\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:09.014395 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.014207 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmfw\" (UniqueName: \"kubernetes.io/projected/cd70c464-1507-414d-96d9-a138e4143281-kube-api-access-hhmfw\") pod \"payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c\" (UID: \"cd70c464-1507-414d-96d9-a138e4143281\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:09.022043 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.022011 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq"] Oct 30 00:46:09.115684 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.115550 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd70c464-1507-414d-96d9-a138e4143281-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c\" (UID: \"cd70c464-1507-414d-96d9-a138e4143281\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:09.115684 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.115640 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmfw\" (UniqueName: \"kubernetes.io/projected/cd70c464-1507-414d-96d9-a138e4143281-kube-api-access-hhmfw\") pod \"payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c\" (UID: \"cd70c464-1507-414d-96d9-a138e4143281\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:09.115902 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.115708 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ce4978-b441-48c9-9145-24c6bdf788f6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq\" (UID: \"65ce4978-b441-48c9-9145-24c6bdf788f6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:09.115902 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.115757 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4c2\" (UniqueName: \"kubernetes.io/projected/65ce4978-b441-48c9-9145-24c6bdf788f6-kube-api-access-pb4c2\") pod \"payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq\" (UID: \"65ce4978-b441-48c9-9145-24c6bdf788f6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:09.116361 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.116341 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd70c464-1507-414d-96d9-a138e4143281-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c\" (UID: \"cd70c464-1507-414d-96d9-a138e4143281\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:09.145805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.145767 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmfw\" (UniqueName: \"kubernetes.io/projected/cd70c464-1507-414d-96d9-a138e4143281-kube-api-access-hhmfw\") pod \"payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c\" (UID: \"cd70c464-1507-414d-96d9-a138e4143281\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:09.180686 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.180647 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:09.217263 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.217231 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ce4978-b441-48c9-9145-24c6bdf788f6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq\" (UID: \"65ce4978-b441-48c9-9145-24c6bdf788f6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:09.217438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.217277 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4c2\" (UniqueName: \"kubernetes.io/projected/65ce4978-b441-48c9-9145-24c6bdf788f6-kube-api-access-pb4c2\") pod \"payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq\" (UID: \"65ce4978-b441-48c9-9145-24c6bdf788f6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:09.218401 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.218372 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ce4978-b441-48c9-9145-24c6bdf788f6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq\" (UID: \"65ce4978-b441-48c9-9145-24c6bdf788f6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:09.252100 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.249471 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4c2\" (UniqueName: \"kubernetes.io/projected/65ce4978-b441-48c9-9145-24c6bdf788f6-kube-api-access-pb4c2\") pod \"payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq\" (UID: \"65ce4978-b441-48c9-9145-24c6bdf788f6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:09.310126 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.310084 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:09.495002 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.494971 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c"] Oct 30 00:46:09.497213 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:09.497187 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd70c464_1507_414d_96d9_a138e4143281.slice/crio-7f87d4cf6131302bbd360abedfc186d2ec801a1af368d85993274ac72b1b284f WatchSource:0}: Error finding container 7f87d4cf6131302bbd360abedfc186d2ec801a1af368d85993274ac72b1b284f: Status 404 returned error can't find the container with id 7f87d4cf6131302bbd360abedfc186d2ec801a1af368d85993274ac72b1b284f Oct 30 00:46:09.623473 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:09.623442 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq"] Oct 30 00:46:09.625854 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:09.625819 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ce4978_b441_48c9_9145_24c6bdf788f6.slice/crio-6dd5bee9c21dacc0a7174542ba0c411d725cbdc73fdbd340c72efcb8a92da78a WatchSource:0}: Error finding container 6dd5bee9c21dacc0a7174542ba0c411d725cbdc73fdbd340c72efcb8a92da78a: Status 404 returned error can't find the container with id 6dd5bee9c21dacc0a7174542ba0c411d725cbdc73fdbd340c72efcb8a92da78a Oct 30 00:46:10.276519 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.276477 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" event={"ID":"65ce4978-b441-48c9-9145-24c6bdf788f6","Type":"ContainerStarted","Data":"ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7"} Oct 30 00:46:10.276988 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.276550 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:10.276988 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.276597 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" event={"ID":"65ce4978-b441-48c9-9145-24c6bdf788f6","Type":"ContainerStarted","Data":"6dd5bee9c21dacc0a7174542ba0c411d725cbdc73fdbd340c72efcb8a92da78a"} Oct 30 00:46:10.278287 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.278259 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" event={"ID":"cd70c464-1507-414d-96d9-a138e4143281","Type":"ContainerStarted","Data":"bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985"} Oct 30 00:46:10.278465 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.278307 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" event={"ID":"cd70c464-1507-414d-96d9-a138e4143281","Type":"ContainerStarted","Data":"7f87d4cf6131302bbd360abedfc186d2ec801a1af368d85993274ac72b1b284f"} Oct 30 00:46:10.278465 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.278346 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:10.317478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.317418 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" podStartSLOduration=2.317403675 podStartE2EDuration="2.317403675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:10.314636299 +0000 UTC m=+708.886562523" watchObservedRunningTime="2026-10-30 00:46:10.317403675 +0000 UTC m=+708.889329897" Oct 30 00:46:10.354174 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.354118 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" podStartSLOduration=2.354103803 podStartE2EDuration="2.354103803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:10.353974061 +0000 UTC m=+708.925900283" watchObservedRunningTime="2026-10-30 00:46:10.354103803 +0000 UTC m=+708.926030024" Oct 30 00:46:10.613781 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.613698 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86"] Oct 30 00:46:10.617562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.617543 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:10.635757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.635724 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-6add35\"" Oct 30 00:46:10.640217 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.640182 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86"] Oct 30 00:46:10.683123 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.683073 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-6add35-dockercfg-mczn6\"" Oct 30 00:46:10.698511 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.698474 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd"] Oct 30 00:46:10.703227 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.703198 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:10.721807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.721773 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd"] Oct 30 00:46:10.732478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.732448 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8cg\" (UniqueName: \"kubernetes.io/projected/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-kube-api-access-wm8cg\") pod \"payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86\" (UID: \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:10.732677 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.732605 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86\" (UID: \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:10.833507 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.833471 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8cg\" (UniqueName: \"kubernetes.io/projected/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-kube-api-access-wm8cg\") pod \"payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86\" (UID: \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:10.833749 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.833547 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da654-992f-48cc-80d2-5b069ed77feb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd\" (UID: \"789da654-992f-48cc-80d2-5b069ed77feb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:10.833749 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.833589 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4m56\" (UniqueName: \"kubernetes.io/projected/789da654-992f-48cc-80d2-5b069ed77feb-kube-api-access-f4m56\") pod \"payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd\" (UID: \"789da654-992f-48cc-80d2-5b069ed77feb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:10.833749 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.833702 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86\" (UID: \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:10.834456 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.834435 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86\" (UID: \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:10.856121 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.856080 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8cg\" (UniqueName: \"kubernetes.io/projected/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-kube-api-access-wm8cg\") pod \"payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86\" (UID: \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:10.932666 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.932631 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:10.934994 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.934963 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da654-992f-48cc-80d2-5b069ed77feb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd\" (UID: \"789da654-992f-48cc-80d2-5b069ed77feb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:10.935142 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.935008 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4m56\" (UniqueName: \"kubernetes.io/projected/789da654-992f-48cc-80d2-5b069ed77feb-kube-api-access-f4m56\") pod \"payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd\" (UID: \"789da654-992f-48cc-80d2-5b069ed77feb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:10.936084 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.935715 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da654-992f-48cc-80d2-5b069ed77feb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd\" (UID: \"789da654-992f-48cc-80d2-5b069ed77feb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:10.992778 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:10.992750 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4m56\" (UniqueName: \"kubernetes.io/projected/789da654-992f-48cc-80d2-5b069ed77feb-kube-api-access-f4m56\") pod \"payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd\" (UID: \"789da654-992f-48cc-80d2-5b069ed77feb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:11.020795 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.020761 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:11.240426 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.240390 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86"] Oct 30 00:46:11.242476 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:11.242434 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7e9d02_c97b_4bca_8bd7_11a746ccbafa.slice/crio-4ac996b700cef866cdf4f6b6b2cc9d60a1e11d18d01762f65f0d7dfd20bcfaec WatchSource:0}: Error finding container 4ac996b700cef866cdf4f6b6b2cc9d60a1e11d18d01762f65f0d7dfd20bcfaec: Status 404 returned error can't find the container with id 4ac996b700cef866cdf4f6b6b2cc9d60a1e11d18d01762f65f0d7dfd20bcfaec Oct 30 00:46:11.289878 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.289611 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" event={"ID":"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa","Type":"ContainerStarted","Data":"4ac996b700cef866cdf4f6b6b2cc9d60a1e11d18d01762f65f0d7dfd20bcfaec"} Oct 30 00:46:11.325933 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.325903 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd"] Oct 30 00:46:11.327844 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:11.327813 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789da654_992f_48cc_80d2_5b069ed77feb.slice/crio-12248254692d72e520a21a63b1464a914fbd6c3fe16c4ca9280ac4f717fb2118 WatchSource:0}: Error finding container 12248254692d72e520a21a63b1464a914fbd6c3fe16c4ca9280ac4f717fb2118: Status 404 returned error can't find the container with id 12248254692d72e520a21a63b1464a914fbd6c3fe16c4ca9280ac4f717fb2118 Oct 30 00:46:11.392669 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.392632 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz"] Oct 30 00:46:11.396410 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.396384 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.413766 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.413724 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-40ac41-serving-cert\"" Oct 30 00:46:11.413976 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.413842 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m"] Oct 30 00:46:11.418686 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.418664 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:11.426954 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.426608 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz"] Oct 30 00:46:11.438637 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.438557 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-beb8d5\"" Oct 30 00:46:11.440658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.438747 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m"] Oct 30 00:46:11.443635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.442375 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-beb8d5-dockercfg-g2fh5\"" Oct 30 00:46:11.468844 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.468815 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr"] Oct 30 00:46:11.472627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.472606 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:11.498781 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.498708 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr"] Oct 30 00:46:11.543017 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.542982 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brn9p\" (UniqueName: \"kubernetes.io/projected/3540ce90-64ec-436a-9c82-21dd2cc2901a-kube-api-access-brn9p\") pod \"payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr\" (UID: \"3540ce90-64ec-436a-9c82-21dd2cc2901a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:11.543206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.543025 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkg8r\" (UniqueName: \"kubernetes.io/projected/269e1e40-2de3-4384-8d31-282bfb066435-kube-api-access-tkg8r\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.543206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.543134 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-maas-api-tls\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.543206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.543166 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4670c74-4555-438a-bdb2-b8d9eafb1a19-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m\" (UID: \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:11.543206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.543188 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e1e40-2de3-4384-8d31-282bfb066435-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.543404 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.543285 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-metrics-certs\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.543404 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.543342 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhn7v\" (UniqueName: \"kubernetes.io/projected/c4670c74-4555-438a-bdb2-b8d9eafb1a19-kube-api-access-dhn7v\") pod \"payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m\" (UID: \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:11.543505 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.543429 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3540ce90-64ec-436a-9c82-21dd2cc2901a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr\" (UID: \"3540ce90-64ec-436a-9c82-21dd2cc2901a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:11.644081 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.644041 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brn9p\" (UniqueName: \"kubernetes.io/projected/3540ce90-64ec-436a-9c82-21dd2cc2901a-kube-api-access-brn9p\") pod \"payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr\" (UID: \"3540ce90-64ec-436a-9c82-21dd2cc2901a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:11.644081 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.644076 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkg8r\" (UniqueName: \"kubernetes.io/projected/269e1e40-2de3-4384-8d31-282bfb066435-kube-api-access-tkg8r\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.644352 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.644115 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-maas-api-tls\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.644352 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.644144 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4670c74-4555-438a-bdb2-b8d9eafb1a19-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m\" (UID: \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:11.644352 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.644173 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e1e40-2de3-4384-8d31-282bfb066435-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.644352 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.644213 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-metrics-certs\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.644352 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.644270 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhn7v\" (UniqueName: \"kubernetes.io/projected/c4670c74-4555-438a-bdb2-b8d9eafb1a19-kube-api-access-dhn7v\") pod \"payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m\" (UID: \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:11.644352 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.644302 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3540ce90-64ec-436a-9c82-21dd2cc2901a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr\" (UID: \"3540ce90-64ec-436a-9c82-21dd2cc2901a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:11.645256 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.645202 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e1e40-2de3-4384-8d31-282bfb066435-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.645687 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.645369 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3540ce90-64ec-436a-9c82-21dd2cc2901a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr\" (UID: \"3540ce90-64ec-436a-9c82-21dd2cc2901a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:11.646010 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.645986 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4670c74-4555-438a-bdb2-b8d9eafb1a19-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m\" (UID: \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:11.647638 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.647610 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-maas-api-tls\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.647948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.647928 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-metrics-certs\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.667443 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.667405 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkg8r\" (UniqueName: \"kubernetes.io/projected/269e1e40-2de3-4384-8d31-282bfb066435-kube-api-access-tkg8r\") pod \"maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.667707 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.667618 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhn7v\" (UniqueName: \"kubernetes.io/projected/c4670c74-4555-438a-bdb2-b8d9eafb1a19-kube-api-access-dhn7v\") pod \"payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m\" (UID: \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:11.668535 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.668503 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brn9p\" (UniqueName: \"kubernetes.io/projected/3540ce90-64ec-436a-9c82-21dd2cc2901a-kube-api-access-brn9p\") pod \"payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr\" (UID: \"3540ce90-64ec-436a-9c82-21dd2cc2901a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:11.710787 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.710752 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:11.738465 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.738422 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:11.788040 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:11.788017 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:12.022775 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.022750 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz"] Oct 30 00:46:12.024356 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:12.024328 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269e1e40_2de3_4384_8d31_282bfb066435.slice/crio-ac817ed143143b7084fcb7ccd3c04a2f89919e3395701f86a57325261d8de5aa WatchSource:0}: Error finding container ac817ed143143b7084fcb7ccd3c04a2f89919e3395701f86a57325261d8de5aa: Status 404 returned error can't find the container with id ac817ed143143b7084fcb7ccd3c04a2f89919e3395701f86a57325261d8de5aa Oct 30 00:46:12.025782 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.025763 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:46:12.060246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.060219 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m"] Oct 30 00:46:12.062793 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:12.062762 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4670c74_4555_438a_bdb2_b8d9eafb1a19.slice/crio-c2a878129f4d373aa395014524ffcdf49308af9b2d98dab4977a345156ced173 WatchSource:0}: Error finding container c2a878129f4d373aa395014524ffcdf49308af9b2d98dab4977a345156ced173: Status 404 returned error can't find the container with id c2a878129f4d373aa395014524ffcdf49308af9b2d98dab4977a345156ced173 Oct 30 00:46:12.108239 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.108216 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr"] Oct 30 00:46:12.110806 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:12.110780 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3540ce90_64ec_436a_9c82_21dd2cc2901a.slice/crio-32042c5e6d7e9a01f05a89acd4c3f29539e672c48117b2e28b4fe100d7ee4bd3 WatchSource:0}: Error finding container 32042c5e6d7e9a01f05a89acd4c3f29539e672c48117b2e28b4fe100d7ee4bd3: Status 404 returned error can't find the container with id 32042c5e6d7e9a01f05a89acd4c3f29539e672c48117b2e28b4fe100d7ee4bd3 Oct 30 00:46:12.297297 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.297183 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" event={"ID":"3540ce90-64ec-436a-9c82-21dd2cc2901a","Type":"ContainerStarted","Data":"af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b"} Oct 30 00:46:12.297297 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.297261 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" event={"ID":"3540ce90-64ec-436a-9c82-21dd2cc2901a","Type":"ContainerStarted","Data":"32042c5e6d7e9a01f05a89acd4c3f29539e672c48117b2e28b4fe100d7ee4bd3"} Oct 30 00:46:12.297297 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.297290 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:12.298833 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.298802 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" event={"ID":"789da654-992f-48cc-80d2-5b069ed77feb","Type":"ContainerStarted","Data":"eae39647c935f34c51c0be9ece233978850257cec10d101966142e1303efda9a"} Oct 30 00:46:12.298958 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.298862 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" event={"ID":"789da654-992f-48cc-80d2-5b069ed77feb","Type":"ContainerStarted","Data":"12248254692d72e520a21a63b1464a914fbd6c3fe16c4ca9280ac4f717fb2118"} Oct 30 00:46:12.298958 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.298891 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:12.300473 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.300452 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" event={"ID":"c4670c74-4555-438a-bdb2-b8d9eafb1a19","Type":"ContainerStarted","Data":"c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5"} Oct 30 00:46:12.300594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.300490 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:12.300594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.300511 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" event={"ID":"c4670c74-4555-438a-bdb2-b8d9eafb1a19","Type":"ContainerStarted","Data":"c2a878129f4d373aa395014524ffcdf49308af9b2d98dab4977a345156ced173"} Oct 30 00:46:12.301689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.301664 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" event={"ID":"269e1e40-2de3-4384-8d31-282bfb066435","Type":"ContainerStarted","Data":"ac817ed143143b7084fcb7ccd3c04a2f89919e3395701f86a57325261d8de5aa"} Oct 30 00:46:12.303101 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.303081 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" event={"ID":"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa","Type":"ContainerStarted","Data":"68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66"} Oct 30 00:46:12.303210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.303200 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:12.336432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.336378 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" podStartSLOduration=1.336358677 podStartE2EDuration="1.336358677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:12.334826194 +0000 UTC m=+710.906752418" watchObservedRunningTime="2026-10-30 00:46:12.336358677 +0000 UTC m=+710.908284898" Oct 30 00:46:12.375103 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.375047 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" podStartSLOduration=2.37502896 podStartE2EDuration="2.37502896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:12.372060272 +0000 UTC m=+710.943986495" watchObservedRunningTime="2026-10-30 00:46:12.37502896 +0000 UTC m=+710.946955221" Oct 30 00:46:12.410314 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.410260 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" podStartSLOduration=2.410245484 podStartE2EDuration="2.410245484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:12.409842849 +0000 UTC m=+710.981769094" watchObservedRunningTime="2026-10-30 00:46:12.410245484 +0000 UTC m=+710.982171706" Oct 30 00:46:12.506167 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:12.506117 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" podStartSLOduration=1.506099876 podStartE2EDuration="1.506099876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:12.498422881 +0000 UTC m=+711.070349104" watchObservedRunningTime="2026-10-30 00:46:12.506099876 +0000 UTC m=+711.078026115" Oct 30 00:46:13.310653 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.310613 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" event={"ID":"269e1e40-2de3-4384-8d31-282bfb066435","Type":"ContainerStarted","Data":"dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1"} Oct 30 00:46:13.311206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.310806 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:13.352146 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.352081 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" podStartSLOduration=1.887843851 podStartE2EDuration="2.352061733s" totalImagesPullingTime="464.217882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:13.351356166 +0000 UTC m=+711.923282388" watchObservedRunningTime="2026-10-30 00:46:13.352061733 +0000 UTC m=+711.923987959" Oct 30 00:46:13.544127 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.544071 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj"] Oct 30 00:46:13.548263 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.548238 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.569882 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.569801 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-6add35-serving-cert\"" Oct 30 00:46:13.572525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.572497 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj"] Oct 30 00:46:13.676977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.676933 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.677172 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.677150 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-metrics-certs\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.677322 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.677299 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9676b8e-5fb3-438d-8138-b6f906d3dd52-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.677405 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.677368 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njk5x\" (UniqueName: \"kubernetes.io/projected/f9676b8e-5fb3-438d-8138-b6f906d3dd52-kube-api-access-njk5x\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.742052 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.742012 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq"] Oct 30 00:46:13.746310 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.746285 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:13.768323 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.768293 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-5062f6\"" Oct 30 00:46:13.772700 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.772667 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq"] Oct 30 00:46:13.778717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.778683 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-metrics-certs\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.778892 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.778775 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9676b8e-5fb3-438d-8138-b6f906d3dd52-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.778892 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.778804 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njk5x\" (UniqueName: \"kubernetes.io/projected/f9676b8e-5fb3-438d-8138-b6f906d3dd52-kube-api-access-njk5x\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.778892 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.778862 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.779913 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.779882 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9676b8e-5fb3-438d-8138-b6f906d3dd52-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.781457 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.781432 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-metrics-certs\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.781847 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.781826 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.802427 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.802387 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njk5x\" (UniqueName: \"kubernetes.io/projected/f9676b8e-5fb3-438d-8138-b6f906d3dd52-kube-api-access-njk5x\") pod \"maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.814395 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.814365 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm"] Oct 30 00:46:13.818631 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.818612 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:13.836070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.836040 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-5062f6-dockercfg-p27sp\"" Oct 30 00:46:13.840107 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.840073 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm"] Oct 30 00:46:13.869663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.869625 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:13.889104 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.889058 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbaf19a5-5ea9-4353-80e8-333668ef5bde-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq\" (UID: \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:13.889284 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.889170 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwrp7\" (UniqueName: \"kubernetes.io/projected/dbaf19a5-5ea9-4353-80e8-333668ef5bde-kube-api-access-pwrp7\") pod \"payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq\" (UID: \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:13.990537 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.990503 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbaf19a5-5ea9-4353-80e8-333668ef5bde-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq\" (UID: \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:13.990712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.990581 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nt8k\" (UniqueName: \"kubernetes.io/projected/ba09d4f5-5e04-4733-8e95-65527ab7ca15-kube-api-access-8nt8k\") pod \"payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm\" (UID: \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:13.990712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.990694 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwrp7\" (UniqueName: \"kubernetes.io/projected/dbaf19a5-5ea9-4353-80e8-333668ef5bde-kube-api-access-pwrp7\") pod \"payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq\" (UID: \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:13.990846 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.990781 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba09d4f5-5e04-4733-8e95-65527ab7ca15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm\" (UID: \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:13.991273 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:13.991253 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbaf19a5-5ea9-4353-80e8-333668ef5bde-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq\" (UID: \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:14.016153 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.016121 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwrp7\" (UniqueName: \"kubernetes.io/projected/dbaf19a5-5ea9-4353-80e8-333668ef5bde-kube-api-access-pwrp7\") pod \"payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq\" (UID: \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:14.060554 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.060517 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:14.092300 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.092218 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nt8k\" (UniqueName: \"kubernetes.io/projected/ba09d4f5-5e04-4733-8e95-65527ab7ca15-kube-api-access-8nt8k\") pod \"payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm\" (UID: \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:14.092300 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.092293 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba09d4f5-5e04-4733-8e95-65527ab7ca15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm\" (UID: \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:14.093304 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.093280 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba09d4f5-5e04-4733-8e95-65527ab7ca15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm\" (UID: \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:14.115873 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.115663 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nt8k\" (UniqueName: \"kubernetes.io/projected/ba09d4f5-5e04-4733-8e95-65527ab7ca15-kube-api-access-8nt8k\") pod \"payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm\" (UID: \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:14.136978 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.136943 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:14.201607 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.200908 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj"] Oct 30 00:46:14.209194 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:14.209115 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9676b8e_5fb3_438d_8138_b6f906d3dd52.slice/crio-db1abdbeaf9a26a3e8e1cba6936edfe8d6a1d703bf99d0671fdd88b077b39e03 WatchSource:0}: Error finding container db1abdbeaf9a26a3e8e1cba6936edfe8d6a1d703bf99d0671fdd88b077b39e03: Status 404 returned error can't find the container with id db1abdbeaf9a26a3e8e1cba6936edfe8d6a1d703bf99d0671fdd88b077b39e03 Oct 30 00:46:14.318525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.318479 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" event={"ID":"f9676b8e-5fb3-438d-8138-b6f906d3dd52","Type":"ContainerStarted","Data":"db1abdbeaf9a26a3e8e1cba6936edfe8d6a1d703bf99d0671fdd88b077b39e03"} Oct 30 00:46:14.371872 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.371825 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq"] Oct 30 00:46:14.373273 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:14.373201 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbaf19a5_5ea9_4353_80e8_333668ef5bde.slice/crio-a5d03d83ecaed080e0e4d55ff391e70c8ecf5bbc7af7c4f0881be686a19310fa WatchSource:0}: Error finding container a5d03d83ecaed080e0e4d55ff391e70c8ecf5bbc7af7c4f0881be686a19310fa: Status 404 returned error can't find the container with id a5d03d83ecaed080e0e4d55ff391e70c8ecf5bbc7af7c4f0881be686a19310fa Oct 30 00:46:14.446692 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.446659 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm"] Oct 30 00:46:14.451722 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:14.451696 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba09d4f5_5e04_4733_8e95_65527ab7ca15.slice/crio-927c43e87c2f5a24b823d1d654c22eee4a62f72de6dfe461ec4704f9ff003b3d WatchSource:0}: Error finding container 927c43e87c2f5a24b823d1d654c22eee4a62f72de6dfe461ec4704f9ff003b3d: Status 404 returned error can't find the container with id 927c43e87c2f5a24b823d1d654c22eee4a62f72de6dfe461ec4704f9ff003b3d Oct 30 00:46:14.621717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.621633 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92"] Oct 30 00:46:14.626048 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.626024 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:14.641769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.641741 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b3593a\"" Oct 30 00:46:14.642668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.642642 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b3593a-dockercfg-dl8qx\"" Oct 30 00:46:14.645397 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.645370 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92"] Oct 30 00:46:14.674777 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.674740 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb"] Oct 30 00:46:14.679752 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.679723 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:14.695646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.695602 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb"] Oct 30 00:46:14.804602 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.804536 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5tr\" (UniqueName: \"kubernetes.io/projected/538ea5e7-d6f3-4391-97f6-4f7389bddb83-kube-api-access-dq5tr\") pod \"payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb\" (UID: \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:14.804794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.804609 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538ea5e7-d6f3-4391-97f6-4f7389bddb83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb\" (UID: \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:14.804794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.804718 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92\" (UID: \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:14.804794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.804768 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lvk\" (UniqueName: \"kubernetes.io/projected/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-kube-api-access-r6lvk\") pod \"payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92\" (UID: \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:14.906126 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.906077 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lvk\" (UniqueName: \"kubernetes.io/projected/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-kube-api-access-r6lvk\") pod \"payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92\" (UID: \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:14.906350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.906219 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5tr\" (UniqueName: \"kubernetes.io/projected/538ea5e7-d6f3-4391-97f6-4f7389bddb83-kube-api-access-dq5tr\") pod \"payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb\" (UID: \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:14.906350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.906254 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538ea5e7-d6f3-4391-97f6-4f7389bddb83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb\" (UID: \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:14.906350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.906279 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92\" (UID: \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:14.907179 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.907157 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92\" (UID: \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:14.907230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.907195 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538ea5e7-d6f3-4391-97f6-4f7389bddb83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb\" (UID: \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:14.931025 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.930995 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lvk\" (UniqueName: \"kubernetes.io/projected/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-kube-api-access-r6lvk\") pod \"payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92\" (UID: \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:14.931185 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.931075 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5tr\" (UniqueName: \"kubernetes.io/projected/538ea5e7-d6f3-4391-97f6-4f7389bddb83-kube-api-access-dq5tr\") pod \"payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb\" (UID: \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:14.940844 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.940816 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:14.998598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:14.997728 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:15.247509 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.247480 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92"] Oct 30 00:46:15.250181 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:15.250153 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2677740b_3783_4ff1_8ff8_7c05c34a7ab5.slice/crio-2a69ec5e1cfb0e7ef88e5634a89361bc982118cda573bad444757db86fbb090c WatchSource:0}: Error finding container 2a69ec5e1cfb0e7ef88e5634a89361bc982118cda573bad444757db86fbb090c: Status 404 returned error can't find the container with id 2a69ec5e1cfb0e7ef88e5634a89361bc982118cda573bad444757db86fbb090c Oct 30 00:46:15.308784 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.308759 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb"] Oct 30 00:46:15.310565 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:15.310531 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538ea5e7_d6f3_4391_97f6_4f7389bddb83.slice/crio-6c0831ea8fc3521e3c0c8402481b09d8a1c00593e68e484f30f0f5e616b7b0a0 WatchSource:0}: Error finding container 6c0831ea8fc3521e3c0c8402481b09d8a1c00593e68e484f30f0f5e616b7b0a0: Status 404 returned error can't find the container with id 6c0831ea8fc3521e3c0c8402481b09d8a1c00593e68e484f30f0f5e616b7b0a0 Oct 30 00:46:15.352060 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.352015 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" event={"ID":"2677740b-3783-4ff1-8ff8-7c05c34a7ab5","Type":"ContainerStarted","Data":"2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85"} Oct 30 00:46:15.352940 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.352093 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" event={"ID":"2677740b-3783-4ff1-8ff8-7c05c34a7ab5","Type":"ContainerStarted","Data":"2a69ec5e1cfb0e7ef88e5634a89361bc982118cda573bad444757db86fbb090c"} Oct 30 00:46:15.355697 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.355541 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" event={"ID":"ba09d4f5-5e04-4733-8e95-65527ab7ca15","Type":"ContainerStarted","Data":"d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a"} Oct 30 00:46:15.355697 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.355615 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" event={"ID":"ba09d4f5-5e04-4733-8e95-65527ab7ca15","Type":"ContainerStarted","Data":"927c43e87c2f5a24b823d1d654c22eee4a62f72de6dfe461ec4704f9ff003b3d"} Oct 30 00:46:15.355884 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.355819 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:15.360772 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.360740 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" event={"ID":"dbaf19a5-5ea9-4353-80e8-333668ef5bde","Type":"ContainerStarted","Data":"3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527"} Oct 30 00:46:15.360980 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.360957 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:15.361111 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.360992 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" event={"ID":"dbaf19a5-5ea9-4353-80e8-333668ef5bde","Type":"ContainerStarted","Data":"a5d03d83ecaed080e0e4d55ff391e70c8ecf5bbc7af7c4f0881be686a19310fa"} Oct 30 00:46:15.363106 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.363067 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" event={"ID":"538ea5e7-d6f3-4391-97f6-4f7389bddb83","Type":"ContainerStarted","Data":"6c0831ea8fc3521e3c0c8402481b09d8a1c00593e68e484f30f0f5e616b7b0a0"} Oct 30 00:46:15.366520 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.366489 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" event={"ID":"f9676b8e-5fb3-438d-8138-b6f906d3dd52","Type":"ContainerStarted","Data":"4ac6609dbab63ead464e209be9cbfb402e8652ddc80314c0f74cd5036d11e86a"} Oct 30 00:46:15.366672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.366646 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:15.395136 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.395102 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr"] Oct 30 00:46:15.399472 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.399444 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.406294 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.406238 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" podStartSLOduration=2.406220364 podStartE2EDuration="2.406220364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:15.398169534 +0000 UTC m=+713.970095757" watchObservedRunningTime="2026-10-30 00:46:15.406220364 +0000 UTC m=+713.978146587" Oct 30 00:46:15.433803 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.429013 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-beb8d5-serving-cert\"" Oct 30 00:46:15.438666 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.436722 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr"] Oct 30 00:46:15.444639 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.444528 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" podStartSLOduration=2.13260782 podStartE2EDuration="2.444514816s" totalImagesPullingTime="311.906996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:15.443908425 +0000 UTC m=+714.015834648" watchObservedRunningTime="2026-10-30 00:46:15.444514816 +0000 UTC m=+714.016441032" Oct 30 00:46:15.484587 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.484521 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" podStartSLOduration=2.484506281 podStartE2EDuration="2.484506281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:15.482182061 +0000 UTC m=+714.054108284" watchObservedRunningTime="2026-10-30 00:46:15.484506281 +0000 UTC m=+714.056432503" Oct 30 00:46:15.515272 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.515178 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-metrics-certs\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.515272 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.515248 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr444\" (UniqueName: \"kubernetes.io/projected/bbef5382-412e-4b85-989e-d728b15c9c72-kube-api-access-sr444\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.515517 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.515363 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-maas-api-tls\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.515517 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.515452 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbef5382-412e-4b85-989e-d728b15c9c72-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.620636 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.617485 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-metrics-certs\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.622290 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.622256 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr444\" (UniqueName: \"kubernetes.io/projected/bbef5382-412e-4b85-989e-d728b15c9c72-kube-api-access-sr444\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.622461 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.622342 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-maas-api-tls\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.622461 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.622450 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbef5382-412e-4b85-989e-d728b15c9c72-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.623380 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.623356 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbef5382-412e-4b85-989e-d728b15c9c72-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.626131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.626099 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-metrics-certs\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.626237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.626141 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-maas-api-tls\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.648247 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.648199 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr444\" (UniqueName: \"kubernetes.io/projected/bbef5382-412e-4b85-989e-d728b15c9c72-kube-api-access-sr444\") pod \"maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:15.719029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:15.718979 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:16.031255 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.031226 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr"] Oct 30 00:46:16.033448 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:16.033421 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbef5382_412e_4b85_989e_d728b15c9c72.slice/crio-9062c397a9b5cb97cd5cfe486f84be7e604fc026a9d42e482550ac6adb34b010 WatchSource:0}: Error finding container 9062c397a9b5cb97cd5cfe486f84be7e604fc026a9d42e482550ac6adb34b010: Status 404 returned error can't find the container with id 9062c397a9b5cb97cd5cfe486f84be7e604fc026a9d42e482550ac6adb34b010 Oct 30 00:46:16.375600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.375544 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" event={"ID":"538ea5e7-d6f3-4391-97f6-4f7389bddb83","Type":"ContainerStarted","Data":"6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087"} Oct 30 00:46:16.375983 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.375631 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:16.377481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.377443 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" event={"ID":"bbef5382-412e-4b85-989e-d728b15c9c72","Type":"ContainerStarted","Data":"9062c397a9b5cb97cd5cfe486f84be7e604fc026a9d42e482550ac6adb34b010"} Oct 30 00:46:16.408633 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.408595 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86"] Oct 30 00:46:16.415135 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.415105 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:16.421795 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.421738 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" podStartSLOduration=2.421717735 podStartE2EDuration="2.421717735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:16.414873166 +0000 UTC m=+714.986799388" watchObservedRunningTime="2026-10-30 00:46:16.421717735 +0000 UTC m=+714.993643958" Oct 30 00:46:16.433743 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.432485 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-89a676\"" Oct 30 00:46:16.433743 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.432543 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86"] Oct 30 00:46:16.433743 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.432609 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-89a676-dockercfg-7mcr5\"" Oct 30 00:46:16.461155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.461099 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" podStartSLOduration=2.461081287 podStartE2EDuration="2.461081287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:16.458182888 +0000 UTC m=+715.030109111" watchObservedRunningTime="2026-10-30 00:46:16.461081287 +0000 UTC m=+715.033007534" Oct 30 00:46:16.535897 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.535847 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7mc\" (UniqueName: \"kubernetes.io/projected/74d6a3b2-8f58-435c-9591-a6d5db1069ce-kube-api-access-vb7mc\") pod \"payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86\" (UID: \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:16.536517 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.536350 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d6a3b2-8f58-435c-9591-a6d5db1069ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86\" (UID: \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:16.637262 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.637220 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7mc\" (UniqueName: \"kubernetes.io/projected/74d6a3b2-8f58-435c-9591-a6d5db1069ce-kube-api-access-vb7mc\") pod \"payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86\" (UID: \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:16.637475 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.637309 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d6a3b2-8f58-435c-9591-a6d5db1069ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86\" (UID: \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:16.638004 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.637986 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d6a3b2-8f58-435c-9591-a6d5db1069ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86\" (UID: \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:16.659932 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.659895 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7mc\" (UniqueName: \"kubernetes.io/projected/74d6a3b2-8f58-435c-9591-a6d5db1069ce-kube-api-access-vb7mc\") pod \"payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86\" (UID: \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:16.745771 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:16.745732 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:17.058081 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:17.058054 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86"] Oct 30 00:46:17.060169 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:17.060138 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d6a3b2_8f58_435c_9591_a6d5db1069ce.slice/crio-92fb039498eacbca9d110eded633a1f32806df18532755ad7c7344c5a7c5d150 WatchSource:0}: Error finding container 92fb039498eacbca9d110eded633a1f32806df18532755ad7c7344c5a7c5d150: Status 404 returned error can't find the container with id 92fb039498eacbca9d110eded633a1f32806df18532755ad7c7344c5a7c5d150 Oct 30 00:46:17.393595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:17.393535 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" event={"ID":"bbef5382-412e-4b85-989e-d728b15c9c72","Type":"ContainerStarted","Data":"2a40aff28eb1a2ac47df36691e66c32d5919f8d05b643ebe1ad3067def0e9618"} Oct 30 00:46:17.394523 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:17.394487 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:17.396948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:17.396912 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" event={"ID":"74d6a3b2-8f58-435c-9591-a6d5db1069ce","Type":"ContainerStarted","Data":"6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6"} Oct 30 00:46:17.397059 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:17.396973 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" event={"ID":"74d6a3b2-8f58-435c-9591-a6d5db1069ce","Type":"ContainerStarted","Data":"92fb039498eacbca9d110eded633a1f32806df18532755ad7c7344c5a7c5d150"} Oct 30 00:46:17.397272 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:17.397254 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:17.397595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:17.397556 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:17.438307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:17.438239 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" podStartSLOduration=2.129038891 podStartE2EDuration="2.438220331s" totalImagesPullingTime="309.18144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:17.434780795 +0000 UTC m=+716.006707017" watchObservedRunningTime="2026-10-30 00:46:17.438220331 +0000 UTC m=+716.010146554" Oct 30 00:46:17.475654 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:17.475552 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" podStartSLOduration=1.475533078 podStartE2EDuration="1.475533078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:17.474852152 +0000 UTC m=+716.046778397" watchObservedRunningTime="2026-10-30 00:46:17.475533078 +0000 UTC m=+716.047459301" Oct 30 00:46:19.059742 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.059703 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598"] Oct 30 00:46:19.064158 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.064139 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.067794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.067770 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.067925 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.067812 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-metrics-certs\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.067925 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.067899 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.068026 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.068014 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dwc7\" (UniqueName: \"kubernetes.io/projected/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-kube-api-access-7dwc7\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.077868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.077845 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-5062f6-serving-cert\"" Oct 30 00:46:19.092611 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.092549 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598"] Oct 30 00:46:19.168922 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.168887 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwc7\" (UniqueName: \"kubernetes.io/projected/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-kube-api-access-7dwc7\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.169117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.168953 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.169117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.169005 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-metrics-certs\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.169117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.169052 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.169233 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:46:19.169053 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-serving-cert: secret "maas-api-e2e-worker-w6-5062f6-serving-cert" not found Oct 30 00:46:19.169233 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:46:19.169191 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls podName:a8d33d24-b5e3-4e1d-b039-b14a8ca03432 nodeName:}" failed. No retries permitted until 2026-10-30 00:46:19.669169658 +0000 UTC m=+718.241095859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls") pod "maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" (UID: "a8d33d24-b5e3-4e1d-b039-b14a8ca03432") : secret "maas-api-e2e-worker-w6-5062f6-serving-cert" not found Oct 30 00:46:19.169823 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.169791 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.171487 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.171465 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-metrics-certs\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.192024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.191997 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dwc7\" (UniqueName: \"kubernetes.io/projected/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-kube-api-access-7dwc7\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.325094 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.325007 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 00:46:19.675458 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.675409 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.678048 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.678015 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.678205 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.678165 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:19.992674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:19.992644 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598"] Oct 30 00:46:19.995185 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:19.995152 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d33d24_b5e3_4e1d_b039_b14a8ca03432.slice/crio-4234de34b4ea0c2559a1beb4dd1239c50e44de7571e60adaa7b5e19900ceca1d WatchSource:0}: Error finding container 4234de34b4ea0c2559a1beb4dd1239c50e44de7571e60adaa7b5e19900ceca1d: Status 404 returned error can't find the container with id 4234de34b4ea0c2559a1beb4dd1239c50e44de7571e60adaa7b5e19900ceca1d Oct 30 00:46:20.418552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.418519 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" event={"ID":"a8d33d24-b5e3-4e1d-b039-b14a8ca03432","Type":"ContainerStarted","Data":"4234de34b4ea0c2559a1beb4dd1239c50e44de7571e60adaa7b5e19900ceca1d"} Oct 30 00:46:20.835495 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.835460 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt"] Oct 30 00:46:20.839729 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.839703 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.858731 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.858704 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b3593a-serving-cert\"" Oct 30 00:46:20.872675 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.872635 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt"] Oct 30 00:46:20.888786 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.888747 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j86w\" (UniqueName: \"kubernetes.io/projected/6436b549-407a-473c-9374-23987295bbec-kube-api-access-6j86w\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.888950 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.888880 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-metrics-certs\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.889043 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.888988 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.889100 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.889041 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6436b549-407a-473c-9374-23987295bbec-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.990108 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.990068 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j86w\" (UniqueName: \"kubernetes.io/projected/6436b549-407a-473c-9374-23987295bbec-kube-api-access-6j86w\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.990294 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.990158 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-metrics-certs\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.990294 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.990210 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.990294 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.990241 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6436b549-407a-473c-9374-23987295bbec-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.990910 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.990887 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6436b549-407a-473c-9374-23987295bbec-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.992789 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.992764 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-metrics-certs\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:20.992862 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:20.992822 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:21.012345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:21.012314 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j86w\" (UniqueName: \"kubernetes.io/projected/6436b549-407a-473c-9374-23987295bbec-kube-api-access-6j86w\") pod \"maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:21.153867 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:21.153830 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:21.292773 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:21.292743 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 00:46:21.292948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:21.292824 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 00:46:21.383926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:21.383898 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 00:46:21.428058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:21.427969 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" event={"ID":"a8d33d24-b5e3-4e1d-b039-b14a8ca03432","Type":"ContainerStarted","Data":"ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded"} Oct 30 00:46:21.428447 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:21.428094 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:21.471426 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:21.470939 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" podStartSLOduration=2.106607598 podStartE2EDuration="2.470919785s" totalImagesPullingTime="364.312187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:21.470001087 +0000 UTC m=+720.041927308" watchObservedRunningTime="2026-10-30 00:46:21.470919785 +0000 UTC m=+720.042846010" Oct 30 00:46:21.486056 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:21.486030 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt"] Oct 30 00:46:21.488720 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:21.488689 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6436b549_407a_473c_9374_23987295bbec.slice/crio-c69c71c14b233ec6bfc665956b0057a92fa55718a2df2ee08c16300038fbfa06 WatchSource:0}: Error finding container c69c71c14b233ec6bfc665956b0057a92fa55718a2df2ee08c16300038fbfa06: Status 404 returned error can't find the container with id c69c71c14b233ec6bfc665956b0057a92fa55718a2df2ee08c16300038fbfa06 Oct 30 00:46:22.436752 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:22.436715 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" event={"ID":"6436b549-407a-473c-9374-23987295bbec","Type":"ContainerStarted","Data":"ceb845f283448f3fbe55d9d4875d1935534057eda2ea77dd0ab6f3d3e03079fd"} Oct 30 00:46:22.437235 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:22.436802 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" event={"ID":"6436b549-407a-473c-9374-23987295bbec","Type":"ContainerStarted","Data":"c69c71c14b233ec6bfc665956b0057a92fa55718a2df2ee08c16300038fbfa06"} Oct 30 00:46:22.437235 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:22.436970 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:22.478146 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:22.478088 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" podStartSLOduration=2.115320432 podStartE2EDuration="2.478071634s" totalImagesPullingTime="362.751202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:22.476307533 +0000 UTC m=+721.048233755" watchObservedRunningTime="2026-10-30 00:46:22.478071634 +0000 UTC m=+721.049997858" Oct 30 00:46:23.314836 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:23.314801 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 00:46:23.315049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:23.315021 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 00:46:23.315114 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:23.315073 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 00:46:23.315178 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:23.315157 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 00:46:23.409165 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:23.409137 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 00:46:26.163996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.163960 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6"] Oct 30 00:46:26.168345 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.168322 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.189683 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.189650 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6"] Oct 30 00:46:26.197028 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.196997 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-2b4105-serving-cert\"" Oct 30 00:46:26.257751 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.257713 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.257939 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.257772 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-metrics-certs\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.257939 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.257909 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw49m\" (UniqueName: \"kubernetes.io/projected/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-kube-api-access-dw49m\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.257939 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.257937 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.358983 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.358944 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-metrics-certs\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.359163 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.359014 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw49m\" (UniqueName: \"kubernetes.io/projected/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-kube-api-access-dw49m\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.359163 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.359045 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.359163 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.359139 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.359860 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.359834 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.362604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.362563 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-metrics-certs\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.362702 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.362636 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.381481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.381451 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 00:46:26.381687 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.381520 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 00:46:26.382373 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.382341 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw49m\" (UniqueName: \"kubernetes.io/projected/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-kube-api-access-dw49m\") pod \"maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.482115 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.482023 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:26.793340 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:26.793313 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6"] Oct 30 00:46:26.795491 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:46:26.795457 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7e2e89_9ed5_4ec1_96e6_a37f5cff6683.slice/crio-748a2b2fe84324fb9baee31d00ea16591c04e55912c5c31e93d5a2e603b523f3 WatchSource:0}: Error finding container 748a2b2fe84324fb9baee31d00ea16591c04e55912c5c31e93d5a2e603b523f3: Status 404 returned error can't find the container with id 748a2b2fe84324fb9baee31d00ea16591c04e55912c5c31e93d5a2e603b523f3 Oct 30 00:46:27.401305 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:27.401275 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 00:46:27.442850 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:27.442821 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 00:46:27.493037 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:27.492879 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" event={"ID":"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683","Type":"ContainerStarted","Data":"f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d"} Oct 30 00:46:27.493037 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:27.492977 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" event={"ID":"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683","Type":"ContainerStarted","Data":"748a2b2fe84324fb9baee31d00ea16591c04e55912c5c31e93d5a2e603b523f3"} Oct 30 00:46:27.493037 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:27.493014 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:27.536385 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:27.536310 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" podStartSLOduration=1.125580222 podStartE2EDuration="1.53628657s" totalImagesPullingTime="410.706348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:27.533487607 +0000 UTC m=+726.105413844" watchObservedRunningTime="2026-10-30 00:46:27.53628657 +0000 UTC m=+726.108212797" Oct 30 00:46:28.406437 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:28.406406 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 00:46:28.406867 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:28.406484 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 00:46:28.452224 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:28.452192 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 00:46:33.508821 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:46:33.508789 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 00:46:34.443908 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:46:34.443870 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:47:04.442826 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:47:04.442743 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:47:18.071474 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.071436 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr"] Oct 30 00:47:18.077124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.077095 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.091398 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.091341 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-beb8d5-openshift-default-dockercfg-kjcx7\"" Oct 30 00:47:18.107230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.107192 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr"] Oct 30 00:47:18.159674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.159643 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3762592f-23e3-476a-a79d-29837a453845-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.159674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.159685 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.159985 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.159715 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.159985 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.159856 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3762592f-23e3-476a-a79d-29837a453845-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.159985 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.159887 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.159985 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.159926 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.160201 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.159993 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.160201 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.160029 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb99c\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-kube-api-access-jb99c\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.160201 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.160177 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.228180 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.228142 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l"] Oct 30 00:47:18.232281 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.232263 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.251718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.251683 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6add35-openshift-default-dockercfg-5zrdb\"" Oct 30 00:47:18.256545 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.256512 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l"] Oct 30 00:47:18.261870 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.261837 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.262049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.261909 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3762592f-23e3-476a-a79d-29837a453845-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.262049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.261955 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.262049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.262001 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.262246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.262064 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3762592f-23e3-476a-a79d-29837a453845-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.262246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.262098 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.262246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.262150 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.262246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.262184 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.262246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.262207 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb99c\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-kube-api-access-jb99c\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.262509 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.262494 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.263296 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.263264 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3762592f-23e3-476a-a79d-29837a453845-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.263897 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.263873 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.264075 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.263936 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.264075 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.263964 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.265264 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.265241 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3762592f-23e3-476a-a79d-29837a453845-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.265928 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.265902 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.284840 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.284808 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.285050 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.285029 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb99c\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-kube-api-access-jb99c\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.363880 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.363786 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.363880 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.363831 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtvl\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-kube-api-access-svtvl\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.364102 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.363914 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.364102 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.363995 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.364102 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.364037 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.364226 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.364156 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.364226 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.364189 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.364226 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.364208 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.364329 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.364314 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.392083 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.392042 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:18.467615 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.467378 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.467615 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.467549 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtvl\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-kube-api-access-svtvl\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.467615 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.467614 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.467901 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.467640 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.467901 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.467683 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.467901 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.467789 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.467901 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.467827 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.467901 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.467858 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.468142 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.467937 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.469235 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.468455 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.469235 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.468486 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.469235 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.469047 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.469732 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.469707 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.469844 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.469764 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.471612 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.471588 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.473334 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.473313 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.491714 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.491679 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.494066 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.494033 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtvl\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-kube-api-access-svtvl\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.545531 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.545494 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:18.720066 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.720031 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr"] Oct 30 00:47:18.721169 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:47:18.721122 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3762592f_23e3_476a_a79d_29837a453845.slice/crio-1f3227ec335b09f29cbd1e448a367e5860ac7d45f231bbc0edb3f69b2fa2692d WatchSource:0}: Error finding container 1f3227ec335b09f29cbd1e448a367e5860ac7d45f231bbc0edb3f69b2fa2692d: Status 404 returned error can't find the container with id 1f3227ec335b09f29cbd1e448a367e5860ac7d45f231bbc0edb3f69b2fa2692d Oct 30 00:47:18.723985 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.723941 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:18.724070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.724036 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:18.724106 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.724084 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:18.836476 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.836436 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" event={"ID":"3762592f-23e3-476a-a79d-29837a453845","Type":"ContainerStarted","Data":"44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a"} Oct 30 00:47:18.836692 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.836508 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" event={"ID":"3762592f-23e3-476a-a79d-29837a453845","Type":"ContainerStarted","Data":"1f3227ec335b09f29cbd1e448a367e5860ac7d45f231bbc0edb3f69b2fa2692d"} Oct 30 00:47:18.854337 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.854313 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l"] Oct 30 00:47:18.857453 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:47:18.857414 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3ccf6c_514d_45b4_bc56_287f5fd0b9f4.slice/crio-f790a1eb8972263ec4a5048d0b4250bf16e7a395bd1b203133f1434e49b81b81 WatchSource:0}: Error finding container f790a1eb8972263ec4a5048d0b4250bf16e7a395bd1b203133f1434e49b81b81: Status 404 returned error can't find the container with id f790a1eb8972263ec4a5048d0b4250bf16e7a395bd1b203133f1434e49b81b81 Oct 30 00:47:18.860233 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.860196 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:18.860343 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.860288 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:18.860343 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.860335 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:18.882232 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:18.882170 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podStartSLOduration=0.882153187 podStartE2EDuration="882.153187ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:18.879233255 +0000 UTC m=+777.451159478" watchObservedRunningTime="2026-10-30 00:47:18.882153187 +0000 UTC m=+777.454079409" Oct 30 00:47:19.394547 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:19.394508 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 00:47:19.395198 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:19.395153 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 00:47:19.395631 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:19.395610 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:19.846145 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:19.846051 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" event={"ID":"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4","Type":"ContainerStarted","Data":"f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778"} Oct 30 00:47:19.846145 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:19.846109 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" event={"ID":"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4","Type":"ContainerStarted","Data":"f790a1eb8972263ec4a5048d0b4250bf16e7a395bd1b203133f1434e49b81b81"} Oct 30 00:47:19.897595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:19.896762 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" podStartSLOduration=1.896741089 podStartE2EDuration="1.896741089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:19.894107466 +0000 UTC m=+778.466033688" watchObservedRunningTime="2026-10-30 00:47:19.896741089 +0000 UTC m=+778.468667313" Oct 30 00:47:20.396092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:20.396048 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 00:47:20.396806 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:20.396754 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 00:47:20.546761 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:20.546679 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:20.547636 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:20.547275 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:47:20.547636 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:20.547342 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 00:47:21.396862 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:21.396822 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 00:47:21.397481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:21.397444 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 00:47:21.547135 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:21.546754 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:47:21.547135 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:21.546816 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 00:47:22.393068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:22.393025 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 00:47:22.393370 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:22.393339 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 00:47:22.546155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:22.546106 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:47:22.546691 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:22.546169 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 00:47:23.394595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:23.394040 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 00:47:23.394595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:23.394107 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 00:47:23.546671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:23.546632 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:47:23.547092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:23.546691 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 00:47:24.397608 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:24.397035 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 00:47:24.397608 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:24.397105 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 00:47:24.546909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:24.546870 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:47:24.547308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:24.546927 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 00:47:25.480012 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:25.479967 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:25.481241 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:25.481217 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:25.482476 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:25.482455 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 00:47:25.573845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:25.573816 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:25.574266 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:25.574114 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:25.575069 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:25.575047 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 00:47:34.443625 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:47:34.443558 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:47:47.864917 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:47.864871 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm"] Oct 30 00:47:47.892429 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:47.892377 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm"] Oct 30 00:47:47.892429 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:47.892424 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:47.909155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:47.909125 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-40ac41-openshift-default-dockercfg-kr4bx\"" Oct 30 00:47:48.017064 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.017030 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.017267 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.017159 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.017326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.017274 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nllnp\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-kube-api-access-nllnp\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.017326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.017311 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.017426 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.017351 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.017426 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.017373 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.017514 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.017495 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.017549 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.017520 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.017549 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.017540 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.118471 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.118383 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.118471 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.118466 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.118696 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.118508 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nllnp\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-kube-api-access-nllnp\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.118696 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.118544 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.118696 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.118614 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.118696 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.118645 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.118696 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.118693 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.118960 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.118717 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.118960 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.118746 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.119072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.119004 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.119072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.119047 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.119250 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.119230 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.119250 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.119240 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.119457 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.119436 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.121017 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.120994 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.121242 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.121223 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.140936 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.140897 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.141084 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.140998 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nllnp\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-kube-api-access-nllnp\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.236088 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.235798 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:48.602286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.602253 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm"] Oct 30 00:47:48.604621 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:47:48.604591 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94498ce_f9f6_4586_8324_e1c5cd85c4f4.slice/crio-279f27c31f8810fc2ab9b69a55031aae645fa64cb0b53cc07a5f1d70513c046d WatchSource:0}: Error finding container 279f27c31f8810fc2ab9b69a55031aae645fa64cb0b53cc07a5f1d70513c046d: Status 404 returned error can't find the container with id 279f27c31f8810fc2ab9b69a55031aae645fa64cb0b53cc07a5f1d70513c046d Oct 30 00:47:48.607373 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.607330 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:48.607496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.607429 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:48.607496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:48.607482 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:49.125273 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:49.125225 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" event={"ID":"e94498ce-f9f6-4586-8324-e1c5cd85c4f4","Type":"ContainerStarted","Data":"6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b"} Oct 30 00:47:49.125768 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:49.125337 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" event={"ID":"e94498ce-f9f6-4586-8324-e1c5cd85c4f4","Type":"ContainerStarted","Data":"279f27c31f8810fc2ab9b69a55031aae645fa64cb0b53cc07a5f1d70513c046d"} Oct 30 00:47:49.193342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:49.187697 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" podStartSLOduration=2.187674013 podStartE2EDuration="2.187674013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:49.183197898 +0000 UTC m=+807.755124121" watchObservedRunningTime="2026-10-30 00:47:49.187674013 +0000 UTC m=+807.759600236" Oct 30 00:47:49.236854 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:49.236816 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:49.238377 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:49.238351 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:47:49.238514 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:49.238414 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 00:47:50.237537 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:50.237499 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:47:50.237972 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:50.237610 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 00:47:51.237626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:51.237554 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:47:51.238045 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:51.237645 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 00:47:52.236918 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:52.236880 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:47:52.237132 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:52.236941 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 00:47:53.236991 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:53.236951 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:47:53.237404 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:53.237013 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 00:47:54.236970 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:54.236932 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:47:54.237152 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:54.236992 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 00:47:55.412356 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:55.412317 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:55.413725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:55.413693 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:47:55.415092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:47:55.415065 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 00:48:04.444431 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:48:04.444386 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:48:18.509777 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.509732 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn"] Oct 30 00:48:18.514605 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.514562 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.539725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.539671 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b3593a-openshift-default-dockercfg-knq67\"" Oct 30 00:48:18.544990 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.544953 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn"] Oct 30 00:48:18.655637 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.655556 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.655811 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.655714 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bz8f\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-kube-api-access-5bz8f\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.655896 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.655803 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.655896 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.655854 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.655978 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.655905 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e117e98e-358b-422a-a2ab-0e80be395b68-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.656029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.656008 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.656114 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.656095 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.656149 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.656123 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.656181 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.656153 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e117e98e-358b-422a-a2ab-0e80be395b68-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.717480 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.717442 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj"] Oct 30 00:48:18.721981 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.721955 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.735833 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.735805 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-89a676-openshift-default-dockercfg-2l6sv\"" Oct 30 00:48:18.745167 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.745135 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj"] Oct 30 00:48:18.757866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.757826 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.758156 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758132 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bz8f\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-kube-api-access-5bz8f\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.758355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758326 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.758355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758212 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.758725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758400 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.758725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758476 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e117e98e-358b-422a-a2ab-0e80be395b68-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.758725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758533 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.758725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758634 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.758725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758660 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.758725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758718 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e117e98e-358b-422a-a2ab-0e80be395b68-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.759058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758746 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.759058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.758785 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.759631 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.759610 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e117e98e-358b-422a-a2ab-0e80be395b68-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.760398 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.759693 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.762028 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.761999 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e117e98e-358b-422a-a2ab-0e80be395b68-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.769816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.769789 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.781978 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.781944 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bz8f\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-kube-api-access-5bz8f\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.782132 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.782040 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.829502 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.829466 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:18.859973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.859930 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.859973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.859973 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.860210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.860118 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.860210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.860193 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.860297 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.860268 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkrn\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-kube-api-access-lmkrn\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.860382 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.860359 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.860525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.860422 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.860525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.860505 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.860669 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.860566 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.962221 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962186 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.962221 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962224 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.962629 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962266 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.962629 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962322 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.962629 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962346 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkrn\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-kube-api-access-lmkrn\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.962629 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962409 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.962629 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962446 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.962629 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962476 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.962629 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962518 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.963021 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962703 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.963021 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962805 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.963021 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.962960 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.963242 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.963168 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.963242 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.963175 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.964769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.964744 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.965525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.965508 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.985765 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.985728 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:18.985933 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:18.985844 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkrn\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-kube-api-access-lmkrn\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:19.036307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.036218 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:19.157361 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.157317 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn"] Oct 30 00:48:19.157698 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:48:19.157669 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode117e98e_358b_422a_a2ab_0e80be395b68.slice/crio-18c7b9b9689e5d70b9ef74183dd9780d67902325c49f9bd4790b80d307c5585c WatchSource:0}: Error finding container 18c7b9b9689e5d70b9ef74183dd9780d67902325c49f9bd4790b80d307c5585c: Status 404 returned error can't find the container with id 18c7b9b9689e5d70b9ef74183dd9780d67902325c49f9bd4790b80d307c5585c Oct 30 00:48:19.160209 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.160171 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:19.160311 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.160248 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:19.160311 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.160276 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:19.346436 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.346413 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj"] Oct 30 00:48:19.348361 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:48:19.348328 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e70cbf_7f1b_4507_81ce_a88fff91e343.slice/crio-e15be7a28797aa509b60a992b44958edf116cb318ecca2aa624d88ec69710b12 WatchSource:0}: Error finding container e15be7a28797aa509b60a992b44958edf116cb318ecca2aa624d88ec69710b12: Status 404 returned error can't find the container with id e15be7a28797aa509b60a992b44958edf116cb318ecca2aa624d88ec69710b12 Oct 30 00:48:19.348929 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.348903 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" event={"ID":"e117e98e-358b-422a-a2ab-0e80be395b68","Type":"ContainerStarted","Data":"24925b422f3e7c08743e97771f69e3fdf0696c6f26132aadc40d7ddf6ee3a161"} Oct 30 00:48:19.349025 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.348956 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" event={"ID":"e117e98e-358b-422a-a2ab-0e80be395b68","Type":"ContainerStarted","Data":"18c7b9b9689e5d70b9ef74183dd9780d67902325c49f9bd4790b80d307c5585c"} Oct 30 00:48:19.350805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.350772 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:19.350874 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.350846 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:19.350910 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.350877 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:19.395089 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.395028 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" podStartSLOduration=1.39500728 podStartE2EDuration="1.39500728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:19.391589893 +0000 UTC m=+837.963516110" watchObservedRunningTime="2026-10-30 00:48:19.39500728 +0000 UTC m=+837.966933502" Oct 30 00:48:19.829686 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:19.829631 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:20.359763 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:20.359719 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" event={"ID":"c1e70cbf-7f1b-4507-81ce-a88fff91e343","Type":"ContainerStarted","Data":"76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa"} Oct 30 00:48:20.359915 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:20.359778 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" event={"ID":"c1e70cbf-7f1b-4507-81ce-a88fff91e343","Type":"ContainerStarted","Data":"e15be7a28797aa509b60a992b44958edf116cb318ecca2aa624d88ec69710b12"} Oct 30 00:48:20.832101 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:20.832059 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:48:20.832624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:20.832115 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:48:21.036967 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:21.036934 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:21.038460 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:21.038428 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:48:21.038616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:21.038492 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:48:21.834597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:21.833924 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:48:21.834597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:21.833988 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:48:22.038709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:22.038672 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:48:22.038909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:22.038729 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:48:22.830246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:22.830076 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:48:22.830246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:22.830163 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:48:23.037115 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:23.037080 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:48:23.037508 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:23.037143 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:48:23.830605 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:23.830013 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:48:23.830605 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:23.830096 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:48:24.037595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:24.037468 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:48:24.037595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:24.037543 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:48:24.835595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:24.834743 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:48:24.835595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:24.834821 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:48:25.037587 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:25.037534 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:48:25.037772 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:25.037614 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:48:25.829952 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:25.829911 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:48:25.830159 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:25.829982 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 00:48:26.037140 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:26.037100 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:48:26.037348 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:26.037163 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 00:48:26.834081 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:26.834046 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:26.834491 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:26.834453 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:26.835085 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:26.835066 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 00:48:26.881349 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:26.881290 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podStartSLOduration=8.881275837 podStartE2EDuration="8.881275837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:20.444003659 +0000 UTC m=+839.015929880" watchObservedRunningTime="2026-10-30 00:48:26.881275837 +0000 UTC m=+845.453202059" Oct 30 00:48:27.042707 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:27.042674 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:27.043094 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:27.043071 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:27.043959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:27.043938 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 00:48:34.445957 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:48:34.445861 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:48:48.139644 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.139597 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c"] Oct 30 00:48:48.144136 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.144116 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.157991 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.157963 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5062f6-openshift-default-dockercfg-9bf6d\"" Oct 30 00:48:48.173583 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.173536 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c"] Oct 30 00:48:48.264551 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.264510 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.264551 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.264560 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.264882 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.264694 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.264882 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.264738 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.264882 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.264877 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.264993 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.264920 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.264993 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.264973 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.265055 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.265035 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99lt\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-kube-api-access-s99lt\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.265107 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.265068 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.366847 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.366809 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99lt\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-kube-api-access-s99lt\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367064 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.366865 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367064 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.366942 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367064 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.366975 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367064 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367002 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367064 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367030 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367129 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367191 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367221 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367503 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367539 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367746 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367620 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367776 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.367997 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.367974 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.369849 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.369829 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.369948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.369879 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.389301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.389263 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.389447 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.389389 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99lt\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-kube-api-access-s99lt\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.459276 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.459189 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:48.785565 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.785535 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c"] Oct 30 00:48:48.786360 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:48:48.786323 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da6e5d2_5ca6_4c6a_9ae6_8c12457f3bde.slice/crio-c1d3eda0ea57c6ff11d0f7a7fd58e974461fddff96658a686689180e5635f9dd WatchSource:0}: Error finding container c1d3eda0ea57c6ff11d0f7a7fd58e974461fddff96658a686689180e5635f9dd: Status 404 returned error can't find the container with id c1d3eda0ea57c6ff11d0f7a7fd58e974461fddff96658a686689180e5635f9dd Oct 30 00:48:48.788959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.788921 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:48.789041 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.788995 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:48.789041 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:48.789025 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:49.671412 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:49.671374 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" event={"ID":"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde","Type":"ContainerStarted","Data":"d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1"} Oct 30 00:48:49.671889 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:49.671432 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" event={"ID":"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde","Type":"ContainerStarted","Data":"c1d3eda0ea57c6ff11d0f7a7fd58e974461fddff96658a686689180e5635f9dd"} Oct 30 00:48:49.742392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:49.742328 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" podStartSLOduration=1.742291174 podStartE2EDuration="1.742291174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:49.72799859 +0000 UTC m=+868.299924812" watchObservedRunningTime="2026-10-30 00:48:49.742291174 +0000 UTC m=+868.314217400" Oct 30 00:48:50.460388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:50.460349 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:50.462036 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:50.462008 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:48:50.462191 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:50.462078 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 00:48:51.366656 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.366621 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f"] Oct 30 00:48:51.371386 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.371357 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.419367 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.419329 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f"] Oct 30 00:48:51.460653 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.460610 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:48:51.460830 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.460672 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 00:48:51.520633 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.520555 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264a783-695c-45cd-953b-9cda029a572b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.520816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.520669 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.520816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.520726 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264a783-695c-45cd-953b-9cda029a572b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.520816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.520754 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.520923 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.520867 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4rl\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-kube-api-access-2w4rl\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.520962 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.520922 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.520996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.520987 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.521028 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.521020 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.521061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.521038 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.622887 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.622796 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623051 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.622936 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264a783-695c-45cd-953b-9cda029a572b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623051 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.623028 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623157 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.623090 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4rl\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-kube-api-access-2w4rl\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623297 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.623267 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623553 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.623358 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623693 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.623627 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623693 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.623678 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623802 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.623703 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623802 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.623789 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264a783-695c-45cd-953b-9cda029a572b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.623905 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.623628 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.624101 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.624050 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.624101 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.624086 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.624458 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.624438 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264a783-695c-45cd-953b-9cda029a572b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.625562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.625542 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.625807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.625788 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264a783-695c-45cd-953b-9cda029a572b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.645386 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.645348 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.645516 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.645481 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4rl\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-kube-api-access-2w4rl\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:51.726721 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:51.726683 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:52.112720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.112686 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f"] Oct 30 00:48:52.114319 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.114279 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:52.114415 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.114354 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:52.114415 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.114387 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:52.460109 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.460071 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:48:52.460524 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.460127 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 00:48:52.728655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.725793 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" event={"ID":"2264a783-695c-45cd-953b-9cda029a572b","Type":"ContainerStarted","Data":"cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927"} Oct 30 00:48:52.728655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.725891 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" event={"ID":"2264a783-695c-45cd-953b-9cda029a572b","Type":"ContainerStarted","Data":"9a206a427664db51f1f045e0dab8d5646eb7ce965f78dff348f7b2bf8399263e"} Oct 30 00:48:52.728655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.727239 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:48:52.790496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:52.790424 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" podStartSLOduration=1.790401533 podStartE2EDuration="1.790401533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:52.785439033 +0000 UTC m=+871.357365258" watchObservedRunningTime="2026-10-30 00:48:52.790401533 +0000 UTC m=+871.362327757" Oct 30 00:48:53.460134 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:53.460089 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:48:53.460649 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:53.460157 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 00:48:53.731013 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:53.730924 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:48:53.731013 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:53.730983 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:48:54.459676 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:54.459637 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:48:54.459873 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:54.459700 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 00:48:54.727647 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:54.727138 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:48:54.727647 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:54.727211 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:48:55.460728 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:55.460685 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 00:48:55.460923 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:55.460746 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 00:48:55.728153 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:55.728075 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:48:55.728614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:55.728459 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:48:56.464227 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:56.464193 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:56.464490 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:56.464465 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:56.465389 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:56.465370 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 00:48:56.728144 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:56.728068 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:48:56.728144 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:56.728131 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:48:57.727534 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:57.727493 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:48:57.727761 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:57.727549 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:48:58.727854 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:58.727814 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:48:58.728328 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:58.727889 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:48:59.727534 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:59.727497 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:48:59.727738 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:48:59.727581 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 00:49:01.101595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:01.101143 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:49:01.105049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:01.102698 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:49:01.105049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:01.103922 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 00:49:03.380213 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.380170 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p"] Oct 30 00:49:03.385230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.385207 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.415739 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.415699 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p"] Oct 30 00:49:03.423720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.423689 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.424706 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.423742 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.424870 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.424754 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218629e5-8db6-4edd-91bc-179727432092-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.424870 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.424859 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.425003 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.424943 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.425061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.425000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218629e5-8db6-4edd-91bc-179727432092-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.425061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.425045 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.425146 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.425111 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.425196 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.425145 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rv7q\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-kube-api-access-9rv7q\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.526882 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.526839 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527076 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.526899 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218629e5-8db6-4edd-91bc-179727432092-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527076 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.526968 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527076 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.526989 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527076 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.527016 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218629e5-8db6-4edd-91bc-179727432092-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527076 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.527047 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527362 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.527102 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527362 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.527126 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rv7q\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-kube-api-access-9rv7q\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527362 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.527205 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.527423 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.527457 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.527692 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.527827 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.527798 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.528106 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.528085 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218629e5-8db6-4edd-91bc-179727432092-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.529552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.529528 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.529979 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.529956 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218629e5-8db6-4edd-91bc-179727432092-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.554681 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.554645 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.554858 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.554705 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rv7q\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-kube-api-access-9rv7q\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.641098 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.641015 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p"] Oct 30 00:49:03.646942 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.646914 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.669044 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.669013 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p"] Oct 30 00:49:03.700682 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.700640 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:03.730347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.730314 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.730501 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.730459 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2h6c\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-kube-api-access-p2h6c\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.730551 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.730536 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.730682 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.730670 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.730722 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.730692 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.730722 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.730716 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.730856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.730734 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.730856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.730848 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.730921 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.730881 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.831914 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.831870 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2h6c\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-kube-api-access-p2h6c\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.832139 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.831958 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.832139 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.832096 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.832139 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.832128 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.832316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.832166 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.832316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.832189 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.832316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.832231 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.832316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.832291 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.832530 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.832339 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.833804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.832900 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.833804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.833105 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.833804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.833207 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.833804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.833502 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.833804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.833745 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.835264 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.835236 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.835865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.835851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.854305 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.854275 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2h6c\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-kube-api-access-p2h6c\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.854912 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.854890 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:03.963376 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:03.963286 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:04.033688 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.032630 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p"] Oct 30 00:49:04.036580 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:49:04.035261 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218629e5_8db6_4edd_91bc_179727432092.slice/crio-2b53d2bf77b8d20722ee86a369d0406823a7f8141bb4a373590738dd028d4bf5 WatchSource:0}: Error finding container 2b53d2bf77b8d20722ee86a369d0406823a7f8141bb4a373590738dd028d4bf5: Status 404 returned error can't find the container with id 2b53d2bf77b8d20722ee86a369d0406823a7f8141bb4a373590738dd028d4bf5 Oct 30 00:49:04.038380 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.038342 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:04.038501 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.038413 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:04.038501 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.038444 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:04.279473 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.279438 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p"] Oct 30 00:49:04.280714 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:49:04.280686 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73f1192_fe9b_46bf_91dd_7995e7b270ad.slice/crio-d0f6b066fed11fe0932f4a2a98267f1113f9679e21e265fae67c07b828a2e810 WatchSource:0}: Error finding container d0f6b066fed11fe0932f4a2a98267f1113f9679e21e265fae67c07b828a2e810: Status 404 returned error can't find the container with id d0f6b066fed11fe0932f4a2a98267f1113f9679e21e265fae67c07b828a2e810 Oct 30 00:49:04.283508 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.283472 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:04.283626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.283549 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:04.283626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.283604 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:04.446000 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:49:04.445966 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:49:04.918953 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.918911 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" event={"ID":"b73f1192-fe9b-46bf-91dd-7995e7b270ad","Type":"ContainerStarted","Data":"3a6c89f77701f4e1488ed2699a6ba1a335200c4dfdee67f9c948f70bac51358d"} Oct 30 00:49:04.919178 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.918977 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" event={"ID":"b73f1192-fe9b-46bf-91dd-7995e7b270ad","Type":"ContainerStarted","Data":"d0f6b066fed11fe0932f4a2a98267f1113f9679e21e265fae67c07b828a2e810"} Oct 30 00:49:04.929538 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.928649 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" event={"ID":"218629e5-8db6-4edd-91bc-179727432092","Type":"ContainerStarted","Data":"838f26bc22a2dd80a3c2346ed5671d8d6dab09e8c7f2afa10c13fa9cf16ffe8a"} Oct 30 00:49:04.929538 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.928723 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" event={"ID":"218629e5-8db6-4edd-91bc-179727432092","Type":"ContainerStarted","Data":"2b53d2bf77b8d20722ee86a369d0406823a7f8141bb4a373590738dd028d4bf5"} Oct 30 00:49:04.964272 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:04.964239 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:05.050081 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:05.050008 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podStartSLOduration=2.049995335 podStartE2EDuration="2.049995335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:05.049742108 +0000 UTC m=+883.621668329" watchObservedRunningTime="2026-10-30 00:49:05.049995335 +0000 UTC m=+883.621921620" Oct 30 00:49:05.053038 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:05.052989 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" podStartSLOduration=2.052976929 podStartE2EDuration="2.052976929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:04.993985522 +0000 UTC m=+883.565911758" watchObservedRunningTime="2026-10-30 00:49:05.052976929 +0000 UTC m=+883.624903150" Oct 30 00:49:05.706532 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:05.706490 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:05.707296 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:05.707268 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:49:05.707486 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:05.707458 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 00:49:05.965964 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:05.965882 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:49:05.965964 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:05.965936 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 00:49:06.720596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:06.719735 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:49:06.720596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:06.719826 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 00:49:06.963822 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:06.963785 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:49:06.964035 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:06.963837 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 00:49:07.707596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:07.706715 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:49:07.707596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:07.706784 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 00:49:07.963926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:07.963832 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:49:07.963926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:07.963895 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 00:49:08.707109 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:08.707066 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:49:08.707446 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:08.707413 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 00:49:08.964298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:08.964215 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:49:08.964298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:08.964269 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 00:49:09.714755 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:09.714712 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:49:09.715863 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:09.715810 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 00:49:09.964389 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:09.964350 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:49:09.964809 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:09.964424 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 00:49:10.701859 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:10.701811 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 00:49:10.702088 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:10.701882 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 00:49:10.964843 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:10.964747 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:49:10.964843 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:10.964808 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 00:49:11.760767 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:11.760739 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:11.762037 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:11.762006 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:11.763128 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:11.763099 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 00:49:12.904595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:12.902863 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:12.904595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:12.904491 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:12.908596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:12.906328 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 00:49:19.266382 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.266324 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6"] Oct 30 00:49:19.287478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.287444 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.295164 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.295135 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6"] Oct 30 00:49:19.302311 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.302278 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"openshift-service-ca.crt\"" Oct 30 00:49:19.313105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.313075 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"default-dockercfg-n8wx7\"" Oct 30 00:49:19.313105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.313089 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"distinct-2-w4-6add35-kserve-self-signed-certs\"" Oct 30 00:49:19.313294 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.313188 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"kube-root-ca.crt\"" Oct 30 00:49:19.346743 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.346705 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-home\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.346955 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.346783 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-model-cache\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.346955 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.346908 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-dshm\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.347047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.347034 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tmp-dir\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.347092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.347082 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5bm\" (UniqueName: \"kubernetes.io/projected/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-kube-api-access-ns5bm\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.347134 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.347113 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tls-certs\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.448194 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.448153 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-dshm\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.448524 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.448502 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tmp-dir\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.449068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.449047 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5bm\" (UniqueName: \"kubernetes.io/projected/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-kube-api-access-ns5bm\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.449562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.449540 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tls-certs\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.450359 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.448966 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tmp-dir\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.450997 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.450797 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-home\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.451161 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.450314 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-home\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.451864 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.451430 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-model-cache\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.452045 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.451839 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-model-cache\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.452045 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.451882 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-dshm\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.454818 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.454796 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tls-certs\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.496602 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.494275 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5bm\" (UniqueName: \"kubernetes.io/projected/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-kube-api-access-ns5bm\") pod \"distinct-2-w4-6add35-kserve-75448b58bc-lrms6\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.603878 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.603779 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:19.915514 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:19.915480 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6"] Oct 30 00:49:19.916310 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:49:19.916280 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41cf6621_92c3_49ea_b3d1_4b52ca5d88a7.slice/crio-e099ba88d70ba925a70053db61ad412aaba943240b49f32bf794a53d77def450 WatchSource:0}: Error finding container e099ba88d70ba925a70053db61ad412aaba943240b49f32bf794a53d77def450: Status 404 returned error can't find the container with id e099ba88d70ba925a70053db61ad412aaba943240b49f32bf794a53d77def450 Oct 30 00:49:20.066936 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:20.066900 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" event={"ID":"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7","Type":"ContainerStarted","Data":"1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc"} Oct 30 00:49:20.067086 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:20.066946 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" event={"ID":"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7","Type":"ContainerStarted","Data":"e099ba88d70ba925a70053db61ad412aaba943240b49f32bf794a53d77def450"} Oct 30 00:49:20.109225 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:20.109170 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podStartSLOduration=1.109156107 podStartE2EDuration="1.109156107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:20.105626036 +0000 UTC m=+898.677552255" watchObservedRunningTime="2026-10-30 00:49:20.109156107 +0000 UTC m=+898.681082329" Oct 30 00:49:20.604222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:20.604179 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:21.695909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:21.695870 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:49:21.696416 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:21.696123 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:49:29.615227 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:29.613991 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:29.645494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:29.645439 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:30.151014 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:30.150977 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:49:34.443584 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:49:34.443533 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:49:43.908068 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.908024 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v"] Oct 30 00:49:43.913495 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.913468 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:43.927998 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.927948 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"kube-root-ca.crt\"" Oct 30 00:49:43.933694 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.933628 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v"] Oct 30 00:49:43.939721 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.939687 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"openshift-service-ca.crt\"" Oct 30 00:49:43.939875 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.939777 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"default-dockercfg-wmkj4\"" Oct 30 00:49:43.939940 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.939919 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"distinct-w0-2b4105-kserve-self-signed-certs\"" Oct 30 00:49:43.966063 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.966027 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-home\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:43.966246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.966070 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tmp-dir\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:43.966246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.966126 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-dshm\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:43.966246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.966145 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tls-certs\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:43.966246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.966195 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsk8g\" (UniqueName: \"kubernetes.io/projected/194f5c4e-881f-4fd2-b053-4d394f77cc5e-kube-api-access-dsk8g\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:43.966402 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:43.966256 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-model-cache\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.067554 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.067522 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-dshm\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.067765 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.067561 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tls-certs\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.067765 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.067649 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsk8g\" (UniqueName: \"kubernetes.io/projected/194f5c4e-881f-4fd2-b053-4d394f77cc5e-kube-api-access-dsk8g\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.067765 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.067713 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-model-cache\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.067937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.067792 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-home\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.067937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.067811 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tmp-dir\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.068196 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.068167 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tmp-dir\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.068305 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.068227 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-home\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.068377 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.068302 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-model-cache\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.069952 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.069924 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-dshm\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.070562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.070544 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tls-certs\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.090874 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.090841 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsk8g\" (UniqueName: \"kubernetes.io/projected/194f5c4e-881f-4fd2-b053-4d394f77cc5e-kube-api-access-dsk8g\") pod \"distinct-w0-2b4105-kserve-bc54c9645-78x5v\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.237662 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.235336 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:44.557757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:44.557728 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v"] Oct 30 00:49:44.559296 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:49:44.559266 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod194f5c4e_881f_4fd2_b053_4d394f77cc5e.slice/crio-8093871dccb1cc97491389d588a8fbe3877c6fcf7ee1bb868f6a62e1ef896adf WatchSource:0}: Error finding container 8093871dccb1cc97491389d588a8fbe3877c6fcf7ee1bb868f6a62e1ef896adf: Status 404 returned error can't find the container with id 8093871dccb1cc97491389d588a8fbe3877c6fcf7ee1bb868f6a62e1ef896adf Oct 30 00:49:45.274258 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:45.274217 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" event={"ID":"194f5c4e-881f-4fd2-b053-4d394f77cc5e","Type":"ContainerStarted","Data":"7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5"} Oct 30 00:49:45.274684 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:45.274272 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" event={"ID":"194f5c4e-881f-4fd2-b053-4d394f77cc5e","Type":"ContainerStarted","Data":"8093871dccb1cc97491389d588a8fbe3877c6fcf7ee1bb868f6a62e1ef896adf"} Oct 30 00:49:45.317199 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:45.317140 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podStartSLOduration=2.317097652 podStartE2EDuration="2.317097652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:45.31344253 +0000 UTC m=+923.885368751" watchObservedRunningTime="2026-10-30 00:49:45.317097652 +0000 UTC m=+923.889023875" Oct 30 00:49:46.235838 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:46.235797 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:54.235766 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:54.235725 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:54.249239 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:54.249201 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:49:54.349746 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:49:54.349717 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:50:03.562102 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.562023 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b"] Oct 30 00:50:03.567865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.567830 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.592131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.592070 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b"] Oct 30 00:50:03.664151 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.664116 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.664355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.664167 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d75156-f3c7-4133-8825-93118ecc86b4-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.664355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.664334 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.664480 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.664367 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjnkr\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-kube-api-access-wjnkr\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.664537 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.664522 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.664626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.664551 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.664714 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.664695 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.664754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.664734 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d75156-f3c7-4133-8825-93118ecc86b4-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.664794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.664757 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.694102 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.694065 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw"] Oct 30 00:50:03.698694 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.698655 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.711929 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.711895 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5"] Oct 30 00:50:03.716274 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.716252 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.721961 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.721927 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw"] Oct 30 00:50:03.753996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.748449 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5"] Oct 30 00:50:03.766501 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766468 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.766501 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766504 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjnkr\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-kube-api-access-wjnkr\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.766816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766541 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce3b57-9419-45e1-a7a2-548ea15e545a-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.766816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766583 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.766816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766745 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.766816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766783 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.767025 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766877 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.767025 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766909 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.767025 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766959 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.767025 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.766960 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d75156-f3c7-4133-8825-93118ecc86b4-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.767234 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767055 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.767234 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767066 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.767234 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767153 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.767390 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767246 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.767390 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767274 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.767390 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767315 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.767390 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767388 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.767560 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767441 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d75156-f3c7-4133-8825-93118ecc86b4-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.767560 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767469 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.767560 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767482 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.767692 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767597 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbx5d\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-kube-api-access-rbx5d\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.767732 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.767698 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.768116 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.768094 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d75156-f3c7-4133-8825-93118ecc86b4-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.769746 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.769722 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.770115 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.770093 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d75156-f3c7-4133-8825-93118ecc86b4-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.791426 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.790285 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.791426 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.790954 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjnkr\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-kube-api-access-wjnkr\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.869324 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869230 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.869324 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869280 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.869534 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869371 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbx5d\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-kube-api-access-rbx5d\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.869534 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869419 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce3b57-9419-45e1-a7a2-548ea15e545a-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.869534 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869453 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.869534 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869479 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.869745 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869538 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.869745 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869662 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.869821 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869782 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.869821 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869793 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.869900 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869853 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.869941 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869901 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhmc\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-kube-api-access-hfhmc\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.869990 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869943 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.869990 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.869973 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.870105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.870020 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.870105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.870081 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.870207 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.870101 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.870207 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.870120 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.870313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.870288 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0686fba-808a-40ab-9317-ed3664c7b7f4-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.870313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.870302 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.870475 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.870386 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.870475 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.870399 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce3b57-9419-45e1-a7a2-548ea15e545a-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.870475 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.870418 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.872221 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.872195 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.872347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.872279 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.886674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.886635 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:03.892514 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.892480 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.892859 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.892833 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbx5d\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-kube-api-access-rbx5d\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:03.972720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.972196 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.972720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.972320 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhmc\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-kube-api-access-hfhmc\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.972720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.972363 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.972720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.972410 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.972720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.972466 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0686fba-808a-40ab-9317-ed3664c7b7f4-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.972720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.972522 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.972720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.972556 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.972720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.972676 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.973275 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.972746 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.973947 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.973802 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.975355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.974089 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.975734 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.975687 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0686fba-808a-40ab-9317-ed3664c7b7f4-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.975734 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.975726 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.975888 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.975778 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.976229 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.976206 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.976690 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.976666 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.996863 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.996825 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhmc\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-kube-api-access-hfhmc\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:03.997137 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:03.997111 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:04.014834 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.014794 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:04.033805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.033771 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:04.217549 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.217507 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b"] Oct 30 00:50:04.217726 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:50:04.217680 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d75156_f3c7_4133_8825_93118ecc86b4.slice/crio-c7c7761f6a22b9b1c206dcf5b12868adc40e9640cd8be568f3ea10682b45d78a WatchSource:0}: Error finding container c7c7761f6a22b9b1c206dcf5b12868adc40e9640cd8be568f3ea10682b45d78a: Status 404 returned error can't find the container with id c7c7761f6a22b9b1c206dcf5b12868adc40e9640cd8be568f3ea10682b45d78a Oct 30 00:50:04.220421 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.220387 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:04.220502 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.220458 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:04.220502 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.220486 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:04.348795 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.348756 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw"] Oct 30 00:50:04.361590 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.359103 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:04.361590 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.359252 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:04.361590 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.359332 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:04.372096 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.372064 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5"] Oct 30 00:50:04.376165 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:50:04.376138 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0686fba_808a_40ab_9317_ed3664c7b7f4.slice/crio-9d18b67f55554f19291aa715b76c4338623adadfce9899009a14648ea84f2920 WatchSource:0}: Error finding container 9d18b67f55554f19291aa715b76c4338623adadfce9899009a14648ea84f2920: Status 404 returned error can't find the container with id 9d18b67f55554f19291aa715b76c4338623adadfce9899009a14648ea84f2920 Oct 30 00:50:04.379195 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.379155 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:04.379304 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.379231 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:04.379304 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.379262 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:04.444731 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.444700 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" event={"ID":"12d75156-f3c7-4133-8825-93118ecc86b4","Type":"ContainerStarted","Data":"cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263"} Oct 30 00:50:04.444865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.444763 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" event={"ID":"12d75156-f3c7-4133-8825-93118ecc86b4","Type":"ContainerStarted","Data":"c7c7761f6a22b9b1c206dcf5b12868adc40e9640cd8be568f3ea10682b45d78a"} Oct 30 00:50:04.447813 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.447605 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" event={"ID":"32ce3b57-9419-45e1-a7a2-548ea15e545a","Type":"ContainerStarted","Data":"e7368fc9bf958d3578e03904f1f294daf36df09c7dbe838f555af5461f797243"} Oct 30 00:50:04.449753 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.449696 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" event={"ID":"c0686fba-808a-40ab-9317-ed3664c7b7f4","Type":"ContainerStarted","Data":"9d18b67f55554f19291aa715b76c4338623adadfce9899009a14648ea84f2920"} Oct 30 00:50:04.459604 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:50:04.458917 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:50:04.508041 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.507989 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" podStartSLOduration=1.507970636 podStartE2EDuration="1.507970636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:04.48852352 +0000 UTC m=+943.060449745" watchObservedRunningTime="2026-10-30 00:50:04.507970636 +0000 UTC m=+943.079896856" Oct 30 00:50:04.890725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:04.890679 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:05.543649 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:05.543607 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" event={"ID":"c0686fba-808a-40ab-9317-ed3664c7b7f4","Type":"ContainerStarted","Data":"61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a"} Oct 30 00:50:05.618193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:05.618152 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" event={"ID":"32ce3b57-9419-45e1-a7a2-548ea15e545a","Type":"ContainerStarted","Data":"059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43"} Oct 30 00:50:05.650604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:05.650267 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podStartSLOduration=2.650246762 podStartE2EDuration="2.650246762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:05.627046896 +0000 UTC m=+944.198973130" watchObservedRunningTime="2026-10-30 00:50:05.650246762 +0000 UTC m=+944.222172985" Oct 30 00:50:05.750609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:05.733348 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podStartSLOduration=2.733325408 podStartE2EDuration="2.733325408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:05.695537418 +0000 UTC m=+944.267463640" watchObservedRunningTime="2026-10-30 00:50:05.733325408 +0000 UTC m=+944.305251632" Oct 30 00:50:05.917597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:05.909332 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:50:05.917597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:05.909398 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:50:06.018598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:06.017976 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:06.024597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:06.020711 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:50:06.024597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:06.020777 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:50:06.040609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:06.037466 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:06.040609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:06.037630 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:50:06.040609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:06.037681 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:50:06.890020 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:06.889979 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:50:06.890330 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:06.890294 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:50:07.022097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:07.022051 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:50:07.022764 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:07.022725 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:50:07.042671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:07.042631 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:50:07.042929 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:07.042897 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:50:07.904337 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:07.904297 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:50:07.906002 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:07.905967 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:50:08.021175 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:08.021116 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:50:08.021531 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:08.021498 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:50:08.038080 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:08.038042 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:50:08.038745 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:08.038710 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:50:08.894076 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:08.894033 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:50:08.894401 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:08.894365 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:50:09.016364 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:09.016319 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:50:09.016746 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:09.016705 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:50:09.043760 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:09.043726 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:50:09.044863 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:09.044826 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:50:09.889023 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:09.888978 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:50:09.890781 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:09.890745 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:50:10.020002 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:10.019959 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:50:10.020329 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:10.020293 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:50:10.038554 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:10.038518 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:50:10.038853 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:10.038826 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:50:10.897427 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:10.896962 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 00:50:10.897427 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:10.897026 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 00:50:11.016043 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:11.016002 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:50:11.016337 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:11.016306 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:50:11.035029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:11.034990 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:50:11.047520 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:11.047371 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:50:11.941059 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:11.941024 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:11.943263 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:11.943237 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:11.944638 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:11.944617 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 00:50:12.015133 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:12.015097 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 00:50:12.015344 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:12.015152 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 00:50:12.034655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:12.034615 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:50:12.034825 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:12.034684 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 00:50:13.020614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:13.020563 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:13.021129 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:13.021114 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:13.022044 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:13.022023 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:50:13.038614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:13.038565 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:13.038990 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:13.038971 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:13.039661 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:13.039639 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:50:18.857973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.857933 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf"] Oct 30 00:50:18.868478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.868448 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.906420 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.906377 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.910334 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.910290 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.910965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.910937 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.911232 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.911212 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.911891 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.911872 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2j48\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-kube-api-access-d2j48\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.912108 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.912080 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.912298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.912280 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.912667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.912650 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.912903 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.912885 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:18.947624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:18.945692 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf"] Oct 30 00:50:19.016195 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.016153 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.016378 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.016236 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.016378 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.016289 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.016378 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.016358 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.016489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.016415 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2j48\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-kube-api-access-d2j48\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.016489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.016450 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.016624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.016490 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.016624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.016525 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.016624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.016595 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.017109 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.017079 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.017228 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.017186 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.017525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.017500 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.017917 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.017891 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.018099 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.018076 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.020036 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.019944 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.020966 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.020943 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.039475 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.039435 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2j48\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-kube-api-access-d2j48\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.039886 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.039864 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.209996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.209959 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq"] Oct 30 00:50:19.215808 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.215781 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.227988 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.227955 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:19.232929 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.232906 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2b4105-openshift-default-dockercfg-wzpvv\"" Oct 30 00:50:19.246329 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.246291 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq"] Oct 30 00:50:19.322388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.321508 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.322388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.321598 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.322388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.321765 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpp4t\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-kube-api-access-qpp4t\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.322388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.321820 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c3d5d-27a6-4464-b973-f7be87174fb6-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.322388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.322005 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.322388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.322044 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.322388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.322081 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.322388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.322101 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.322388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.322175 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.424419 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.424490 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.424598 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpp4t\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-kube-api-access-qpp4t\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.424630 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c3d5d-27a6-4464-b973-f7be87174fb6-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.424797 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.424834 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.424870 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.424898 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.424975 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.426592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.425433 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.477676 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.427539 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.477676 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.427848 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.477676 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.428884 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c3d5d-27a6-4464-b973-f7be87174fb6-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.477676 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.456320 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.573608 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.573045 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.774981 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.773441 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpp4t\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-kube-api-access-qpp4t\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.986377 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.986076 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:19.986377 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:19.986324 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:20.133165 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:20.132842 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:20.234990 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:20.234952 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf"] Oct 30 00:50:20.309095 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:20.309044 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:20.309357 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:20.309332 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:20.309515 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:20.309495 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:21.524598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:21.494085 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" event={"ID":"c27bd511-fd29-4697-8cd3-6d9d91ea9228","Type":"ContainerStarted","Data":"da4b60830c859647e114963fd70ddef0a29d34ad68902ed8873e9507452a76e9"} Oct 30 00:50:21.524598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:21.494181 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" event={"ID":"c27bd511-fd29-4697-8cd3-6d9d91ea9228","Type":"ContainerStarted","Data":"3031e49f9064a91c1efd343cb5287536fb0786ec677549e1b3db738fa0faccd6"} Oct 30 00:50:21.787192 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:21.786181 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podStartSLOduration=3.786156797 podStartE2EDuration="3.786156797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:21.579290075 +0000 UTC m=+960.151216298" watchObservedRunningTime="2026-10-30 00:50:21.786156797 +0000 UTC m=+960.358083021" Oct 30 00:50:21.935981 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:21.935932 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:21.936271 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:21.936242 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:21.936419 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:21.936400 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:22.025779 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:22.025713 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq"] Oct 30 00:50:22.230460 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:22.230415 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:22.233001 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:22.231684 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:50:22.233682 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:22.233642 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 00:50:22.629528 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:22.629490 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" event={"ID":"005c3d5d-27a6-4464-b973-f7be87174fb6","Type":"ContainerStarted","Data":"f4d98e63fae6d47c172c86eb24a6a49aef0b0d19d96d67c3bf20b28a46b3ef71"} Oct 30 00:50:23.229372 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:23.229329 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:50:23.229630 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:23.229405 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 00:50:23.651064 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:23.651027 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" event={"ID":"005c3d5d-27a6-4464-b973-f7be87174fb6","Type":"ContainerStarted","Data":"29189591a7ec02f9603d3899f85d86fd18f55b5a815db83b04d6aa16110b2a0d"} Oct 30 00:50:23.736710 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:23.736638 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" podStartSLOduration=4.736616779 podStartE2EDuration="4.736616779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:23.692948589 +0000 UTC m=+962.264874836" watchObservedRunningTime="2026-10-30 00:50:23.736616779 +0000 UTC m=+962.308543002" Oct 30 00:50:24.152242 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:24.149808 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:24.152242 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:24.151469 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:50:24.152242 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:24.151527 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:50:24.235472 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:24.235421 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:50:24.237846 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:24.236646 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 00:50:25.139594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:25.139005 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:50:25.139594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:25.139068 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:50:25.229017 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:25.228980 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:50:25.229222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:25.229089 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 00:50:26.134104 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:26.134062 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:50:26.134408 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:26.134377 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:50:26.229084 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:26.228991 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:50:26.229685 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:26.229648 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 00:50:27.134876 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:27.134834 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:50:27.135060 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:27.134909 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:50:27.232093 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:27.232049 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:50:27.234010 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:27.233974 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 00:50:28.134876 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:28.134838 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:50:28.135205 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:28.135164 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:50:28.231212 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:28.231178 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:50:28.231689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:28.231481 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 00:50:29.134158 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:29.134118 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:50:29.134617 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:29.134190 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:50:29.228374 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:29.228335 4927 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:50:29.228559 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:29.228394 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 00:50:30.135596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:30.134991 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:50:30.135596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:30.135072 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 00:50:30.238696 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:30.237121 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:30.238696 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:30.238471 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:30.239328 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:30.239307 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 00:50:30.987198 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:30.987164 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s"] Oct 30 00:50:30.994082 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:30.994056 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.012061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.012026 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s"] Oct 30 00:50:31.015793 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.015759 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"distinct-2-w0-2b4105-kserve-self-signed-certs\"" Oct 30 00:50:31.075584 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.075540 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-home\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.075794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.075617 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gszj\" (UniqueName: \"kubernetes.io/projected/598fbcd3-e89c-4308-833e-fa9fc2e3251c-kube-api-access-5gszj\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.075794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.075730 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tmp-dir\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.075921 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.075799 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-dshm\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.075921 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.075885 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-model-cache\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.076096 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.076076 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tls-certs\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.139332 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.139297 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:31.139872 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.139702 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:31.140538 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.140517 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 00:50:31.188814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.188768 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tls-certs\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.197715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.192899 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-home\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.197715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.192982 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gszj\" (UniqueName: \"kubernetes.io/projected/598fbcd3-e89c-4308-833e-fa9fc2e3251c-kube-api-access-5gszj\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.197715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.193059 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tmp-dir\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.197715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.193141 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-dshm\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.197715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.193294 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-model-cache\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.197715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.193885 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tmp-dir\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.197715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.194155 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-home\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.197715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.194717 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-model-cache\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.250199 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.250114 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tls-certs\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.260012 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.259977 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-dshm\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.261966 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.261935 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gszj\" (UniqueName: \"kubernetes.io/projected/598fbcd3-e89c-4308-833e-fa9fc2e3251c-kube-api-access-5gszj\") pod \"distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.312758 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.312725 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:31.714077 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.714046 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s"] Oct 30 00:50:31.715662 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:50:31.715627 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod598fbcd3_e89c_4308_833e_fa9fc2e3251c.slice/crio-195333d0a98577babe21e1312bc7eaa615fa168a5159e1fd77a6b72f0a3c1b0b WatchSource:0}: Error finding container 195333d0a98577babe21e1312bc7eaa615fa168a5159e1fd77a6b72f0a3c1b0b: Status 404 returned error can't find the container with id 195333d0a98577babe21e1312bc7eaa615fa168a5159e1fd77a6b72f0a3c1b0b Oct 30 00:50:31.859093 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.859051 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" event={"ID":"598fbcd3-e89c-4308-833e-fa9fc2e3251c","Type":"ContainerStarted","Data":"1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3"} Oct 30 00:50:31.859301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.859103 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" event={"ID":"598fbcd3-e89c-4308-833e-fa9fc2e3251c","Type":"ContainerStarted","Data":"195333d0a98577babe21e1312bc7eaa615fa168a5159e1fd77a6b72f0a3c1b0b"} Oct 30 00:50:31.904427 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:31.904364 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podStartSLOduration=1.904348316 podStartE2EDuration="1.904348316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:31.901566922 +0000 UTC m=+970.473493146" watchObservedRunningTime="2026-10-30 00:50:31.904348316 +0000 UTC m=+970.476274546" Oct 30 00:50:32.314279 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:32.314245 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:34.444225 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:50:34.444184 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:50:41.313876 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:41.313837 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:41.327933 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:41.327901 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:41.961359 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:41.961330 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:46.345953 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:46.345902 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:50:46.821598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:46.821417 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:50:46.960881 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:46.958677 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:47.153756 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:47.152681 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:47.402593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:47.349813 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:50:47.402593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:47.350619 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:50:47.892177 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:47.892120 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:50:48.047595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.035772 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.250720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.250257 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:48.665358 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:50:48.665321 4927 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.002s" Oct 30 00:50:48.674037 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.674003 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:50:48.674323 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.674309 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:48.674443 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.674434 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:50:48.674544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.674535 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:50:48.674665 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.674654 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:50:48.674784 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.674775 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:50:48.674890 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.674881 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:50:48.856795 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.856764 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2"] Oct 30 00:50:48.862195 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.862167 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:48.893279 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.893223 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2"] Oct 30 00:50:48.937887 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.937783 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de24b267-3437-4138-90fa-fbf49824d0a4-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:48.937887 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.937864 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:48.938131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.937905 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:48.938131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.937961 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:48.938131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.937988 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78r67\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-kube-api-access-78r67\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:48.938131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.938034 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:48.938131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.938062 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:48.938131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.938114 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de24b267-3437-4138-90fa-fbf49824d0a4-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:48.938428 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:48.938160 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.039895 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.039856 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.039902 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78r67\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-kube-api-access-78r67\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.039944 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.039971 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040019 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de24b267-3437-4138-90fa-fbf49824d0a4-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040059 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040340 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040300 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de24b267-3437-4138-90fa-fbf49824d0a4-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040395 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040366 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040405 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040410 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040558 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040449 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040558 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040524 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040795 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040772 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.040889 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.040864 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de24b267-3437-4138-90fa-fbf49824d0a4-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.042629 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.042603 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.042993 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.042966 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de24b267-3437-4138-90fa-fbf49824d0a4-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.063135 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.063105 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78r67\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-kube-api-access-78r67\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.064934 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.064910 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.188965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.188862 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:49.525995 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.525963 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2"] Oct 30 00:50:49.526478 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:50:49.526447 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde24b267_3437_4138_90fa_fbf49824d0a4.slice/crio-0758d428fcbf6f7fb39632b6e99df34a3158b18dc134bb79d091dd4fb44fe33e WatchSource:0}: Error finding container 0758d428fcbf6f7fb39632b6e99df34a3158b18dc134bb79d091dd4fb44fe33e: Status 404 returned error can't find the container with id 0758d428fcbf6f7fb39632b6e99df34a3158b18dc134bb79d091dd4fb44fe33e Oct 30 00:50:49.529124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.529089 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:49.529207 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.529167 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:49.529207 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.529196 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:49.760014 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.758416 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" event={"ID":"de24b267-3437-4138-90fa-fbf49824d0a4","Type":"ContainerStarted","Data":"b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84"} Oct 30 00:50:49.760014 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.758490 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" event={"ID":"de24b267-3437-4138-90fa-fbf49824d0a4","Type":"ContainerStarted","Data":"0758d428fcbf6f7fb39632b6e99df34a3158b18dc134bb79d091dd4fb44fe33e"} Oct 30 00:50:49.808084 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:49.807979 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podStartSLOduration=1.807962255 podStartE2EDuration="1.807962255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:49.804739219 +0000 UTC m=+988.376665441" watchObservedRunningTime="2026-10-30 00:50:49.807962255 +0000 UTC m=+988.379888476" Oct 30 00:50:50.189895 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:50.189863 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:50.191356 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:50.191325 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:50:50.191497 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:50.191418 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:50:51.195047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:51.195004 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:50:51.195740 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:51.195706 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:50:52.199285 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:52.199240 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:50:52.199895 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:52.199313 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:50:53.193022 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:53.192981 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:50:53.193337 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:53.193303 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:50:54.194606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:54.191997 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:50:54.194606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:54.192067 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:50:55.195856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:55.195815 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:50:55.196530 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:55.196495 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:50:56.190320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:56.190285 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:50:56.190630 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:56.190346 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 00:50:57.325248 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:57.325199 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:57.325824 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:57.325533 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:57.326926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:57.326828 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:50:58.601070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.601035 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv"] Oct 30 00:50:58.607168 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.607140 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.627298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.627259 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv"] Oct 30 00:50:58.634210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.634179 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"distinct-2-w1-beb8d5-kserve-self-signed-certs\"" Oct 30 00:50:58.634210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.634204 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"default-dockercfg-82c28\"" Oct 30 00:50:58.634448 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.634326 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"openshift-service-ca.crt\"" Oct 30 00:50:58.634448 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.634353 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-beb8d5\"/\"kube-root-ca.crt\"" Oct 30 00:50:58.719839 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.719800 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-home\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.720049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.719855 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-dshm\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.720049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.719963 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-tmp-dir\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.720150 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.720085 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-model-cache\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.720266 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.720244 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wrwr\" (UniqueName: \"kubernetes.io/projected/9c804efc-0218-41c4-9f85-bb362fb167d5-kube-api-access-6wrwr\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.720324 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.720278 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c804efc-0218-41c4-9f85-bb362fb167d5-tls-certs\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.825594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.821674 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-home\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.825594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.821766 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-dshm\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.825594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.821799 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-tmp-dir\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.825594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.821949 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-model-cache\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.825594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.822119 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wrwr\" (UniqueName: \"kubernetes.io/projected/9c804efc-0218-41c4-9f85-bb362fb167d5-kube-api-access-6wrwr\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.825594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.822151 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c804efc-0218-41c4-9f85-bb362fb167d5-tls-certs\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.825594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.824036 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-tmp-dir\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.825594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.824312 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-model-cache\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.831935 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.831900 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-home\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.860505 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.859933 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-dshm\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.874063 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.874022 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c804efc-0218-41c4-9f85-bb362fb167d5-tls-certs\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.887382 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.887339 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wrwr\" (UniqueName: \"kubernetes.io/projected/9c804efc-0218-41c4-9f85-bb362fb167d5-kube-api-access-6wrwr\") pod \"distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:58.925039 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:58.925002 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:59.302305 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:59.302271 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv"] Oct 30 00:50:59.303528 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:50:59.303497 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c804efc_0218_41c4_9f85_bb362fb167d5.slice/crio-5664bb00e5385f98728cbe8c7516699be81c5c29a0e83db4010f586b2ad4d682 WatchSource:0}: Error finding container 5664bb00e5385f98728cbe8c7516699be81c5c29a0e83db4010f586b2ad4d682: Status 404 returned error can't find the container with id 5664bb00e5385f98728cbe8c7516699be81c5c29a0e83db4010f586b2ad4d682 Oct 30 00:50:59.888410 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:59.888365 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" event={"ID":"9c804efc-0218-41c4-9f85-bb362fb167d5","Type":"ContainerStarted","Data":"507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3"} Oct 30 00:50:59.888849 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:59.888423 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" event={"ID":"9c804efc-0218-41c4-9f85-bb362fb167d5","Type":"ContainerStarted","Data":"5664bb00e5385f98728cbe8c7516699be81c5c29a0e83db4010f586b2ad4d682"} Oct 30 00:50:59.926389 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:59.926339 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:50:59.931042 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:50:59.930994 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podStartSLOduration=1.930979483 podStartE2EDuration="1.930979483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:59.92718939 +0000 UTC m=+998.499115600" watchObservedRunningTime="2026-10-30 00:50:59.930979483 +0000 UTC m=+998.502905748" Oct 30 00:51:04.450775 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:51:04.450734 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:51:08.925681 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:08.925640 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:08.938940 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:08.938906 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:08.967858 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:08.967824 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:11.954734 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:11.954678 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:11.961719 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:11.961666 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.308842 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:12.308729 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.327032 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:12.326983 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.824425 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:12.824389 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:12.831624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:12.824696 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.942485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:12.834925 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:12.988598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:12.959941 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:51:12.988598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:12.973725 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:51:12.988598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:12.975026 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.001698 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:12.992461 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:13.152181 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.148450 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.229593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.229027 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:13.229593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.229089 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.231923 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.231779 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:13.231923 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.231842 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.309598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.309508 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.311851 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.311814 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:51:13.464384 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.436671 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:13.464384 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.436745 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.918219 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.918182 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:13.949650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.949606 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:13.950496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.950469 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:51:13.964712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:13.964664 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.075904 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:14.075828 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:14.077097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:14.077054 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.155992 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:14.155951 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:14.156190 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:14.156016 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.325350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:14.314529 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:14.574041 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:14.564219 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 00:51:14.933179 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:14.933148 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:51:14.956250 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:14.956202 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:51:14.967591 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:14.967395 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:51:18.466679 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.466633 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64"] Oct 30 00:51:18.471727 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.471700 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.502727 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.502694 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64"] Oct 30 00:51:18.618950 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.618911 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4a673-85e7-412c-bd86-33d62b2be721-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.619137 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.619023 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.619137 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.619053 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.619137 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.619087 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.619137 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.619103 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4a673-85e7-412c-bd86-33d62b2be721-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.619273 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.619149 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bjc\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-kube-api-access-q8bjc\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.619324 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.619302 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.619369 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.619335 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.619414 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.619401 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.721488 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.721403 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.721650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.721537 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.721650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.721621 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.721837 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.721818 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4a673-85e7-412c-bd86-33d62b2be721-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.721926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.721909 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.721988 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.721938 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.721988 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.721970 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.722118 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.722009 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.722118 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.722026 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.722118 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.722042 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4a673-85e7-412c-bd86-33d62b2be721-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.722118 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.722086 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bjc\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-kube-api-access-q8bjc\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.722318 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.722236 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.722377 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.722346 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.722685 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.722653 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4a673-85e7-412c-bd86-33d62b2be721-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.724326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.724299 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.724661 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.724635 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4a673-85e7-412c-bd86-33d62b2be721-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.747516 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.747479 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.747516 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.747496 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bjc\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-kube-api-access-q8bjc\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:18.793262 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:18.793223 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:19.131964 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.131933 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64"] Oct 30 00:51:19.132871 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:51:19.132847 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a4a673_85e7_412c_bd86_33d62b2be721.slice/crio-1ae0491c4784ebbfafbd1e7e435e3413d48c6898ebd48c3db41eb1ed9bb9c225 WatchSource:0}: Error finding container 1ae0491c4784ebbfafbd1e7e435e3413d48c6898ebd48c3db41eb1ed9bb9c225: Status 404 returned error can't find the container with id 1ae0491c4784ebbfafbd1e7e435e3413d48c6898ebd48c3db41eb1ed9bb9c225 Oct 30 00:51:19.135285 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.135249 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:19.135380 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.135320 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:19.135380 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.135350 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:19.266348 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.266312 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb"] Oct 30 00:51:19.271160 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.271135 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.283743 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.283714 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk"] Oct 30 00:51:19.288496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.288467 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.297649 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.296965 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb"] Oct 30 00:51:19.333443 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.333404 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c47f328-0846-4b77-b15e-d4561701475f-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.333740 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.333718 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.333975 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.333950 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c47f328-0846-4b77-b15e-d4561701475f-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.334053 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334026 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.334120 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334084 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.334177 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334119 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.334228 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334173 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.334228 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334200 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.334331 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334248 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.334331 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334281 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.334435 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334357 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.334499 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334438 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.334553 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334505 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.334670 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334547 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.334670 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334613 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg8tc\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-kube-api-access-rg8tc\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.334781 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.334740 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.335108 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.335081 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.335188 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.335129 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnzx2\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-kube-api-access-nnzx2\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.356591 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.350613 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk"] Oct 30 00:51:19.436374 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436282 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzx2\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-kube-api-access-nnzx2\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.436374 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436356 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c47f328-0846-4b77-b15e-d4561701475f-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.436374 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436376 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.436726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436393 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c47f328-0846-4b77-b15e-d4561701475f-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.436726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436425 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.436726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436453 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.436726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436490 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.436726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436534 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.436726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436564 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.436726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436618 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.436726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436666 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436743 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436819 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.437117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436892 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436922 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436954 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg8tc\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-kube-api-access-rg8tc\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436960 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.437117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.436980 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.437117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.437093 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.437606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.437194 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.437282 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.437347 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.437439 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.437565 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.437903 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.437698 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.437903 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.437851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.438016 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.437914 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.438424 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.438402 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c47f328-0846-4b77-b15e-d4561701475f-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.439750 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.439727 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.439982 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.439954 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.440106 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.440086 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.440167 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.440123 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c47f328-0846-4b77-b15e-d4561701475f-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.472751 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.472711 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w"] Oct 30 00:51:19.480065 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.480018 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.481322 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.480680 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.481322 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.480686 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg8tc\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-kube-api-access-rg8tc\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.481322 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.481274 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnzx2\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-kube-api-access-nnzx2\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.482800 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.482756 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.534671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.534323 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w"] Oct 30 00:51:19.538736 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.538690 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.538907 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.538867 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4316fd2e-adab-4e5e-9b8d-c139100038a4-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.538971 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.538907 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.538971 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.538950 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.539078 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.538990 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.539078 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.539041 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.539078 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.539067 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.539240 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.539161 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rmzd\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-kube-api-access-5rmzd\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.539240 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.539222 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.590512 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.590473 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:19.606554 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.606450 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:19.641173 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.641124 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.641340 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.641308 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.641490 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.641473 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4316fd2e-adab-4e5e-9b8d-c139100038a4-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.641533 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.641519 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.641591 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.641559 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.641640 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.641626 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.641818 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.641679 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.641818 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.641720 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.641818 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.641798 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rmzd\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-kube-api-access-5rmzd\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.643158 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.643126 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.643158 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.643153 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.643448 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.643429 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.643581 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.643544 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.643965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.643947 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4316fd2e-adab-4e5e-9b8d-c139100038a4-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.644873 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.644844 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.646047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.646020 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.670461 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.670432 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rmzd\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-kube-api-access-5rmzd\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.686204 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.683152 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.810596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.810522 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:19.941730 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:51:19.941675 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8008c2_15fb_4bf0_80e2_575a4f60abe6.slice/crio-1ec3aa4ed3d88dde6892ca2d091b664e3be0922cc7a4d210c6ec6b99fc667d34 WatchSource:0}: Error finding container 1ec3aa4ed3d88dde6892ca2d091b664e3be0922cc7a4d210c6ec6b99fc667d34: Status 404 returned error can't find the container with id 1ec3aa4ed3d88dde6892ca2d091b664e3be0922cc7a4d210c6ec6b99fc667d34 Oct 30 00:51:19.942538 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.942511 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb"] Oct 30 00:51:19.950549 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.949863 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:19.950549 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.949977 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:19.950549 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.950062 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:19.993283 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:19.993240 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk"] Oct 30 00:51:20.024756 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.024713 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" event={"ID":"3b8008c2-15fb-4bf0-80e2-575a4f60abe6","Type":"ContainerStarted","Data":"1ec3aa4ed3d88dde6892ca2d091b664e3be0922cc7a4d210c6ec6b99fc667d34"} Oct 30 00:51:20.030514 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.030478 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" event={"ID":"83a4a673-85e7-412c-bd86-33d62b2be721","Type":"ContainerStarted","Data":"de62e63e2098732cf7d2bc7429da8e0342c0c15c7ab3a1cd241c8d43b576c3ac"} Oct 30 00:51:20.030740 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.030533 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" event={"ID":"83a4a673-85e7-412c-bd86-33d62b2be721","Type":"ContainerStarted","Data":"1ae0491c4784ebbfafbd1e7e435e3413d48c6898ebd48c3db41eb1ed9bb9c225"} Oct 30 00:51:20.036416 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:51:20.035886 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c47f328_0846_4b77_b15e_d4561701475f.slice/crio-1a443c36a67259dbee57d5228366e5645dd38169bf94dfa57ab7d7fe9e30e80c WatchSource:0}: Error finding container 1a443c36a67259dbee57d5228366e5645dd38169bf94dfa57ab7d7fe9e30e80c: Status 404 returned error can't find the container with id 1a443c36a67259dbee57d5228366e5645dd38169bf94dfa57ab7d7fe9e30e80c Oct 30 00:51:20.038797 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.038757 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:20.038984 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.038957 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:20.039091 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.039008 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:20.114951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.114770 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podStartSLOduration=2.114749847 podStartE2EDuration="2.114749847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:20.073625303 +0000 UTC m=+1018.645551526" watchObservedRunningTime="2026-10-30 00:51:20.114749847 +0000 UTC m=+1018.686676072" Oct 30 00:51:20.239155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.234248 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w"] Oct 30 00:51:20.260761 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.253285 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:20.261367 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.261323 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:20.261546 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.261515 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:20.806097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.806006 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:20.830598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.828929 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:51:20.830598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:20.829012 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:51:21.048298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.048254 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" event={"ID":"4316fd2e-adab-4e5e-9b8d-c139100038a4","Type":"ContainerStarted","Data":"4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc"} Oct 30 00:51:21.048603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.048561 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" event={"ID":"4316fd2e-adab-4e5e-9b8d-c139100038a4","Type":"ContainerStarted","Data":"aafd360d302980b1472965201f3a43c64ed7077d3b935c545584bf1557d703d4"} Oct 30 00:51:21.051752 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.051719 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" event={"ID":"6c47f328-0846-4b77-b15e-d4561701475f","Type":"ContainerStarted","Data":"e7f04844e1d2d39ab1931c2efb4b792e1bd889f47f601721509818ae0135dc60"} Oct 30 00:51:21.052363 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.051773 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" event={"ID":"6c47f328-0846-4b77-b15e-d4561701475f","Type":"ContainerStarted","Data":"1a443c36a67259dbee57d5228366e5645dd38169bf94dfa57ab7d7fe9e30e80c"} Oct 30 00:51:21.056230 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.056139 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" event={"ID":"3b8008c2-15fb-4bf0-80e2-575a4f60abe6","Type":"ContainerStarted","Data":"9d784856571772591add28df8e0b90b090c921d7c694fb0895b1b5f430dacd5c"} Oct 30 00:51:21.298594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.283770 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podStartSLOduration=2.283732442 podStartE2EDuration="2.283732442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:21.253495531 +0000 UTC m=+1019.825421756" watchObservedRunningTime="2026-10-30 00:51:21.283732442 +0000 UTC m=+1019.855658666" Oct 30 00:51:21.298594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.284501 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podStartSLOduration=2.284486283 podStartE2EDuration="2.284486283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:21.16732694 +0000 UTC m=+1019.739253162" watchObservedRunningTime="2026-10-30 00:51:21.284486283 +0000 UTC m=+1019.856412505" Oct 30 00:51:21.322656 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.315609 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podStartSLOduration=2.306553696 podStartE2EDuration="2.306553696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:21.303190576 +0000 UTC m=+1019.875116799" watchObservedRunningTime="2026-10-30 00:51:21.306553696 +0000 UTC m=+1019.878479920" Oct 30 00:51:21.846596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.826288 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:21.949769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.884342 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:51:21.949769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.884431 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:51:21.949769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.909285 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:51:21.949769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.909341 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:51:21.949769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.921727 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:51:21.949769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:21.921792 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:51:22.185662 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.124365 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:22.185662 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.124444 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:22.185662 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.124710 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:51:22.185662 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.124773 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 00:51:22.592246 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.592138 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:51:22.598203 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.598162 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:51:22.623276 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.623182 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:51:22.623472 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.623275 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 00:51:22.842651 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.827016 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:51:22.842651 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:22.827090 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:51:23.030594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.000682 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:51:23.069762 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.042224 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:23.119812 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.119689 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:23.256933 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.256894 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:51:23.273056 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.257254 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:51:23.351604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.351065 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:23.483595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.449361 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:51:23.648773 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.648731 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:51:23.649053 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.649021 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 00:51:23.667110 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.667073 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:51:23.667405 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.667375 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:51:23.929488 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.929449 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:51:23.935211 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.935167 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:51:23.935494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.929503 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:51:23.935669 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:23.935644 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:51:24.017597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.011033 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:24.017597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.011528 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:51:24.128031 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.127979 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:24.175596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.152604 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:24.202657 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.177001 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:24.202657 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.177460 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:51:24.403642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.359304 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:24.403642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.359799 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:51:24.552656 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.552265 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 00:51:24.625607 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.613301 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:51:24.625607 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.613360 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:51:24.649601 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.645480 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:51:24.649601 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.645549 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 00:51:24.888868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.860522 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:51:24.888868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.860605 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:51:24.926515 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.924092 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:51:24.926515 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:24.924157 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:51:25.016735 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.016524 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:25.177663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.176657 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:25.189598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.185259 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:51:25.241642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.240960 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:25.364601 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.363726 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:25.606761 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.606558 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:51:25.607410 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.607337 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:51:25.608740 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.608708 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:51:25.608992 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.608906 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 00:51:25.804593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.800952 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:51:25.804593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.801016 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:51:25.823596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.813732 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:51:25.823596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:25.813811 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:51:26.022729 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.022671 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:26.190596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.174155 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:26.190596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.180668 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:26.237599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.236523 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:26.379661 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.374729 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:26.595741 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.595081 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:51:26.595741 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.595147 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:51:26.645076 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.644970 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:51:26.646681 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.646642 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 00:51:26.882237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.881714 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:51:26.882237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.881783 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:51:26.882237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.882231 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:51:26.882633 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:26.882266 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:51:27.026180 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.026069 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:27.179433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.178357 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:27.179433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.179268 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:51:27.184370 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.183983 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:27.294868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.282873 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:27.377932 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.376778 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:27.754592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.656752 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:51:27.754592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.656816 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 00:51:27.754592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.657409 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:51:27.754592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.657445 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:51:27.811710 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.811665 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 00:51:27.812002 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.811967 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 00:51:27.812777 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.812747 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:51:27.812969 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:27.812944 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:51:28.004463 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.004420 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:51:28.061695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.051920 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:28.223769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.223706 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:28.324647 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.316841 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:51:28.324647 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.322654 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:28.384707 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.383716 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:51:28.625657 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.617110 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:51:28.625657 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.617250 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:51:28.625657 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.618010 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:51:28.625657 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.618080 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 00:51:28.663489 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.663455 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:28.854267 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.853407 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:51:28.854267 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.853474 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:51:28.865688 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.864725 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:51:28.957986 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:28.957873 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:51:29.250415 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:29.249618 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:29.251370 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:29.251316 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:29.255150 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:29.255070 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 00:51:29.592913 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:29.592819 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 00:51:29.592913 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:29.592878 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 00:51:29.609359 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:29.609315 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:51:29.609618 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:29.609403 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 00:51:29.812594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:29.812385 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 00:51:29.812594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:29.812467 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 00:51:30.598398 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:30.598363 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:30.599186 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:30.599152 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:30.600041 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:30.600016 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:51:30.611782 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:30.611756 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:30.612085 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:30.612066 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:30.612972 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:30.612951 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:51:31.007330 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:31.007291 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:31.007998 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:31.007971 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:31.008775 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:31.008752 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 00:51:34.320599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.320497 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z"] Oct 30 00:51:34.326433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.326408 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.366024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.365991 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z"] Oct 30 00:51:34.370432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.370401 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.370624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.370446 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.370624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.370565 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.370719 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.370669 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.370719 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.370704 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.370790 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.370762 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.370831 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.370796 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkwv\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-kube-api-access-rzkwv\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.370867 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.370859 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.370909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.370875 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.415803 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.415764 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8"] Oct 30 00:51:34.424128 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.424099 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.461142 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:51:34.461100 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:51:34.464451 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.464408 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8"] Oct 30 00:51:34.472721 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.472690 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.472901 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.472735 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkwv\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-kube-api-access-rzkwv\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.472901 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.472813 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.472901 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.472839 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.473034 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.472920 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.473092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473057 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.473217 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473195 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.473354 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473324 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c22e25-5694-4368-9462-5b84f2226eb4-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.473482 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473350 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.473482 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473212 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.473482 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473394 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjp8z\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-kube-api-access-zjp8z\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.473689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473560 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.473689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473626 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.473689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473665 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.473842 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473709 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.473842 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473769 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.473842 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473835 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.474006 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473863 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c22e25-5694-4368-9462-5b84f2226eb4-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.474006 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473901 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.474006 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.473942 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.474674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.474646 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.475705 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.475682 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.476133 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.476106 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.476262 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.476124 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.476942 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.476919 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.497300 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.497264 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.497820 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.497796 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkwv\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-kube-api-access-rzkwv\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.575983 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.575885 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjp8z\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-kube-api-access-zjp8z\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576165 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576055 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576165 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576102 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576285 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576192 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576285 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576239 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c22e25-5694-4368-9462-5b84f2226eb4-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576424 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576409 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576488 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576434 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576488 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576470 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576651 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576602 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576651 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576625 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576838 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576632 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c22e25-5694-4368-9462-5b84f2226eb4-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576838 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576814 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.576939 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.576851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.577054 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.577027 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c22e25-5694-4368-9462-5b84f2226eb4-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.578866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.578838 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.579043 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.579026 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c22e25-5694-4368-9462-5b84f2226eb4-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.598735 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.598697 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.598867 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.598828 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjp8z\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-kube-api-access-zjp8z\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.643947 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.643909 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:34.756253 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.756215 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:34.981355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.981312 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z"] Oct 30 00:51:34.981553 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:51:34.981524 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a9b83a_6f6f_44e3_81b8_6e363d9fcaa5.slice/crio-790314ade38637f40fc778b59b50a40fa47bb0cd9ab5fda1c9d69e410735c03d WatchSource:0}: Error finding container 790314ade38637f40fc778b59b50a40fa47bb0cd9ab5fda1c9d69e410735c03d: Status 404 returned error can't find the container with id 790314ade38637f40fc778b59b50a40fa47bb0cd9ab5fda1c9d69e410735c03d Oct 30 00:51:34.984232 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.984196 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:34.984318 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.984268 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:34.984318 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:34.984298 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:35.089997 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:51:35.089963 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c22e25_5694_4368_9462_5b84f2226eb4.slice/crio-f60918405e9301931dff838a91ee2cd8a0d0b132bc5e72eccea1826aefa29853 WatchSource:0}: Error finding container f60918405e9301931dff838a91ee2cd8a0d0b132bc5e72eccea1826aefa29853: Status 404 returned error can't find the container with id f60918405e9301931dff838a91ee2cd8a0d0b132bc5e72eccea1826aefa29853 Oct 30 00:51:35.091339 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.091310 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8"] Oct 30 00:51:35.093234 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.093193 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:35.093309 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.093290 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:35.093352 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.093342 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:35.507732 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.507689 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" event={"ID":"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5","Type":"ContainerStarted","Data":"da5e491af6571aa496e21a52b86556f7ccaa99788fe288a24a30d0757cf8a7ec"} Oct 30 00:51:35.508194 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.507758 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" event={"ID":"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5","Type":"ContainerStarted","Data":"790314ade38637f40fc778b59b50a40fa47bb0cd9ab5fda1c9d69e410735c03d"} Oct 30 00:51:35.509651 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.509622 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" event={"ID":"01c22e25-5694-4368-9462-5b84f2226eb4","Type":"ContainerStarted","Data":"1e8f32a8ec256628dcc592493a9680815c534411eab738ba38c4b83aafb55183"} Oct 30 00:51:35.509791 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.509668 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" event={"ID":"01c22e25-5694-4368-9462-5b84f2226eb4","Type":"ContainerStarted","Data":"f60918405e9301931dff838a91ee2cd8a0d0b132bc5e72eccea1826aefa29853"} Oct 30 00:51:35.553530 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.553470 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podStartSLOduration=1.553448794 podStartE2EDuration="1.553448794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:35.550328904 +0000 UTC m=+1034.122255127" watchObservedRunningTime="2026-10-30 00:51:35.553448794 +0000 UTC m=+1034.125375018" Oct 30 00:51:35.596094 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.596036 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podStartSLOduration=1.596019946 podStartE2EDuration="1.596019946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:35.593404547 +0000 UTC m=+1034.165330763" watchObservedRunningTime="2026-10-30 00:51:35.596019946 +0000 UTC m=+1034.167946167" Oct 30 00:51:35.644812 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.644776 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:35.762784 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:35.762695 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:36.646482 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:36.646440 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:36.646927 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:36.646494 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:36.759131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:36.759098 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:36.759324 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:36.759154 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:37.648598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:37.647094 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:37.648598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:37.647250 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:37.768978 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:37.765026 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:37.768978 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:37.765113 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:38.644349 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:38.644309 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:38.644619 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:38.644374 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:38.756830 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:38.756790 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:38.757358 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:38.756859 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:39.644613 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:39.644549 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:39.644833 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:39.644628 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:39.767758 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:39.759837 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:39.767758 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:39.759939 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:40.650308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:40.650244 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:40.651112 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:40.650555 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:40.758078 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:40.758032 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:40.758429 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:40.758385 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:41.654596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:41.654133 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:41.654596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:41.654223 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:41.684540 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:41.684085 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:41.770685 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:41.769131 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:41.770685 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:41.769197 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:41.955078 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:41.954914 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.084598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.075701 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.234801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.234683 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.319730 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.319671 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.369462 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.369411 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.664513 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.664468 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:42.665211 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.665174 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:42.686739 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.686682 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.687298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.687274 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:42.765092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.764163 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:42.765092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.764229 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:42.976688 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.973768 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:42.976688 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:42.974452 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:51:43.091627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.076474 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.091627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.077177 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:51:43.091908 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.091756 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:43.091908 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.091821 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:43.378147 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.376086 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:51:43.378147 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.377590 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:51:43.696494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.681085 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:43.696494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.681162 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:43.806599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.791702 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:43.806599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.791783 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:43.862868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.832793 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:43.997773 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:43.997667 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.196692 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:44.139145 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.292666 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:44.288930 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.379922 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:44.379856 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.583659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:44.555241 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.723317 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:44.656315 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:44.723317 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:44.656392 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:44.723317 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:44.676538 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:44.787332 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:44.787267 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:44.787670 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:44.787628 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:45.241684 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.241604 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:51:45.276700 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.254331 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:51:45.276700 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.264375 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:45.340405 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.340359 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:51:45.451850 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.430215 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:51:45.460607 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.381829 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:45.709625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.709554 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:45.753804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.710005 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:45.753804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.710426 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:45.767796 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.765086 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:45.823114 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.823075 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:45.823397 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:45.823367 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:46.078313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.078192 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" podUID="c19cce81-4a48-4b54-b2ac-29ef57f9395c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 00:51:46.256667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.251330 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.282642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.269681 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.466280 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.466143 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:51:46.500644 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.471910 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.645522 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.645472 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:46.645869 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.645821 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:46.683765 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.683712 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:46.684164 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.684114 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.684480 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.684457 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:46.684644 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.684623 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.713997 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.713943 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:46.762647 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.761563 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:46.762647 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:46.761641 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:47.270630 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:47.262600 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:47.350616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:47.342715 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:51:47.482320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:47.482258 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:47.625680 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:47.586100 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:51:47.799630 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:47.796072 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:47.918659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:47.884894 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 00:51:47.918659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:47.905666 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 00:51:48.039646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:48.039606 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 00:51:48.231932 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:48.231823 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 00:51:48.269603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:48.262932 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:48.346275 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:48.346228 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:48.489387 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:48.487825 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:48.590712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:48.590662 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:48.806663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:48.804169 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:49.320598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.312314 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:49.352469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.351393 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:49.521598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.501500 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:51:49.634751 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.634701 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:49.706853 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.706117 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:49.706853 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.706187 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:49.770758 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.770717 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:51:49.771038 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.771021 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:49.771199 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.771184 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:51:49.772546 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.772527 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:51:49.772838 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.772824 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:49.773095 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.773083 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:51:49.775799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.775781 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:51:49.803918 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.803159 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:49.803918 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:49.803222 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:50.083472 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:50.083378 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:50.538631 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:50.538561 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 00:51:50.751594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:50.750413 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:50.751594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:50.750487 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:51.437326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:51.437291 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:51:51.438966 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:51:51.438936 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:52:03.609437 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.609395 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn"] Oct 30 00:52:03.615963 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.615936 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.650300 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.650262 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn"] Oct 30 00:52:03.732775 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.732737 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327358b-05aa-4bce-8ace-8789224ccbdf-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.732997 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.732784 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhq7k\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-kube-api-access-qhq7k\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.732997 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.732936 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.733129 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.733008 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.733276 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.733248 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.733364 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.733350 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.733424 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.733390 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.733519 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.733496 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.733671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.733655 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.835547 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.835494 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhq7k\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-kube-api-access-qhq7k\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.835774 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.835614 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.835774 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.835663 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.835873 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.835786 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.835873 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.835852 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.835975 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.835902 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.835975 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.835966 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.836079 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.836019 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.836164 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.836141 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.836245 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.836209 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.836302 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.836219 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327358b-05aa-4bce-8ace-8789224ccbdf-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.836302 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.836270 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.836376 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.836297 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.836862 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.836838 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327358b-05aa-4bce-8ace-8789224ccbdf-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.838617 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.838550 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.838741 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.838589 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.858196 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.858156 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhq7k\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-kube-api-access-qhq7k\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.858378 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.858204 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:03.932734 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:03.932702 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:04.267481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:04.267456 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn"] Oct 30 00:52:04.268948 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:52:04.268921 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5327358b_05aa_4bce_8ace_8789224ccbdf.slice/crio-5929f2f47e3730c52fcc628888c4c010a764852f11a4c804f3ca45cc9a1c6038 WatchSource:0}: Error finding container 5929f2f47e3730c52fcc628888c4c010a764852f11a4c804f3ca45cc9a1c6038: Status 404 returned error can't find the container with id 5929f2f47e3730c52fcc628888c4c010a764852f11a4c804f3ca45cc9a1c6038 Oct 30 00:52:04.271700 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:04.271666 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:04.271768 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:04.271739 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:04.271805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:04.271767 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:04.449934 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:52:04.449894 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:52:04.665021 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:04.664977 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" event={"ID":"5327358b-05aa-4bce-8ace-8789224ccbdf","Type":"ContainerStarted","Data":"ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b"} Oct 30 00:52:04.665548 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:04.665053 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" event={"ID":"5327358b-05aa-4bce-8ace-8789224ccbdf","Type":"ContainerStarted","Data":"5929f2f47e3730c52fcc628888c4c010a764852f11a4c804f3ca45cc9a1c6038"} Oct 30 00:52:04.716442 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:04.716374 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podStartSLOduration=1.716357641 podStartE2EDuration="1.716357641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:04.711448517 +0000 UTC m=+1063.283374772" watchObservedRunningTime="2026-10-30 00:52:04.716357641 +0000 UTC m=+1063.288283862" Oct 30 00:52:04.937656 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:04.935681 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:05.942595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:05.935725 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:05.942595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:05.935788 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:06.940598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:06.940022 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:06.940598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:06.940106 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:07.216124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:07.215995 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.254597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:07.242729 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.533657 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:07.532690 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.557604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:07.549868 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:07.801804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:07.801682 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.237687 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.237625 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.268305 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.246364 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.270436 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.270407 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:52:08.270719 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.270702 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:52:08.415598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.411799 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:08.465625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.453786 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:08.543692 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.541895 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:08.568616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.561189 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:52:08.576597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.575133 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:52:08.630600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.577678 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:52:08.867656 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.851398 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:52:08.894366 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:08.853553 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:52:09.025602 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.009249 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:09.025965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.025919 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:09.255610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.254684 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:09.264354 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.264293 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:52:09.560674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.560550 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:09.671630 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.659052 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:52:09.887942 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.887898 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:52:09.955602 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.947220 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:09.955602 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.947285 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:09.955602 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:09.947723 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:10.383663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:10.375856 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:52:10.397599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:10.394272 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:52:10.699655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:10.699028 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:52:10.724598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:10.704709 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Oct 30 00:52:10.783610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:10.772245 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:52:10.903596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:10.894104 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:52:11.321515 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.321469 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:11.321846 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.321817 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:11.484768 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.464508 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:52:11.485480 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.465454 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:52:11.530301 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:52:11.514023 4927 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.95s" Oct 30 00:52:11.549598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.546087 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:11.574797 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.574687 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:11.659597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.640270 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:11.659597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.640343 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:11.717806 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.717368 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:52:11.775450 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.775393 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:52:11.903800 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:11.903746 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.195648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:12.191652 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:12.265611 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:12.258065 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:12.506678 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:12.486962 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.587871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:12.587803 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.616990 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:12.616931 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.771652 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:12.748254 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:52:12.921595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:12.781761 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:12.980595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:12.943047 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:13.076953 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:13.076825 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:13.150666 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:13.016237 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:52:13.164419 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:13.164392 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:52:13.209138 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:13.208435 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:52:13.209619 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:13.209403 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:52:13.212557 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:13.212510 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:52:13.225169 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:13.225127 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:52:13.936226 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:13.933999 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:13.936226 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:13.934058 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:14.933585 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:14.933537 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:14.933781 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:14.933640 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:15.204757 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.204668 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8982-207bf0"] Oct 30 00:52:15.210254 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.210227 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:15.225320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.225290 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8982-207bf0"] Oct 30 00:52:15.230945 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.230921 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v4hbx\"" Oct 30 00:52:15.367003 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.366964 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mmz\" (UniqueName: \"kubernetes.io/projected/116e2909-5e63-418f-8e1e-276816f05036-kube-api-access-79mmz\") pod \"test-curl-8982-207bf0\" (UID: \"116e2909-5e63-418f-8e1e-276816f05036\") " pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:15.469308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.469202 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mmz\" (UniqueName: \"kubernetes.io/projected/116e2909-5e63-418f-8e1e-276816f05036-kube-api-access-79mmz\") pod \"test-curl-8982-207bf0\" (UID: \"116e2909-5e63-418f-8e1e-276816f05036\") " pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:15.534655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.532599 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mmz\" (UniqueName: \"kubernetes.io/projected/116e2909-5e63-418f-8e1e-276816f05036-kube-api-access-79mmz\") pod \"test-curl-8982-207bf0\" (UID: \"116e2909-5e63-418f-8e1e-276816f05036\") " pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:15.826818 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.826721 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:15.934355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.934322 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 00:52:15.934516 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:15.934378 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 00:52:16.163215 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:16.163184 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8982-207bf0"] Oct 30 00:52:16.165089 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:52:16.165052 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod116e2909_5e63_418f_8e1e_276816f05036.slice/crio-5a03e9000817a5615321c070e032ad03da810b3271c428e317097b7b948a33ce WatchSource:0}: Error finding container 5a03e9000817a5615321c070e032ad03da810b3271c428e317097b7b948a33ce: Status 404 returned error can't find the container with id 5a03e9000817a5615321c070e032ad03da810b3271c428e317097b7b948a33ce Oct 30 00:52:16.168155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:16.168134 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:52:16.236510 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:16.236472 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8982-207bf0" event={"ID":"116e2909-5e63-418f-8e1e-276816f05036","Type":"ContainerStarted","Data":"5a03e9000817a5615321c070e032ad03da810b3271c428e317097b7b948a33ce"} Oct 30 00:52:17.544330 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:17.544234 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:17.547970 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:17.547778 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:17.560454 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:17.559192 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 00:52:21.421977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.421940 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8982-207bf0" event={"ID":"116e2909-5e63-418f-8e1e-276816f05036","Type":"ContainerStarted","Data":"27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195"} Oct 30 00:52:21.459798 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.459737 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8982-207bf0" podStartSLOduration=1.680095008 podStartE2EDuration="6.459721838s" totalImagesPullingTime="4.77962683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:21.458091292 +0000 UTC m=+1080.030017513" watchObservedRunningTime="2026-10-30 00:52:21.459721838 +0000 UTC m=+1080.031648060" Oct 30 00:52:21.661618 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.661556 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2"] Oct 30 00:52:21.668022 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.667993 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.688532 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.688489 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2"] Oct 30 00:52:21.692123 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.691894 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1e708b2f-gw-openshift-default-dockercfg-lrxwg\"" Oct 30 00:52:21.721585 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.721527 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-socket\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.721821 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.721613 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-certs\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.721821 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.721666 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-credential-socket\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.721821 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.721699 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-podinfo\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.721989 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.721841 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-envoy\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.721989 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.721961 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-data\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.722099 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.721987 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wb7\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-kube-api-access-j7wb7\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.723114 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.722056 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istiod-ca-cert\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.723294 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.723149 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-token\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.840411 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istiod-ca-cert\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.840467 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-token\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.840588 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-socket\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.840626 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-certs\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.840672 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-credential-socket\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.840699 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-podinfo\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.840834 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-envoy\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.840944 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-data\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.840965 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wb7\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-kube-api-access-j7wb7\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.842594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.842110 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istiod-ca-cert\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.865659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.851464 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-socket\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.865659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.854905 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-certs\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.865659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.855420 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-credential-socket\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.865659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.855549 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-data\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.905596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.905138 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-podinfo\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.927646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.926605 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-envoy\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.971310 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.971210 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-token\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:21.973358 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:21.973325 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wb7\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-kube-api-access-j7wb7\") pod \"e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:22.032897 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:22.031651 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:22.535996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:22.535962 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2"] Oct 30 00:52:22.536365 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:52:22.536224 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3fcea8_9fbb_4cc3_909a_1ca4deb37721.slice/crio-f0f2bec14bcfd20b14ee2c5e162f4cd1f7494d08aa007e8027fb53968415bb17 WatchSource:0}: Error finding container f0f2bec14bcfd20b14ee2c5e162f4cd1f7494d08aa007e8027fb53968415bb17: Status 404 returned error can't find the container with id f0f2bec14bcfd20b14ee2c5e162f4cd1f7494d08aa007e8027fb53968415bb17 Oct 30 00:52:22.539474 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:22.539424 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:22.539607 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:22.539528 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:22.539607 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:22.539593 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:22.630366 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:22.630335 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8982-207bf0"] Oct 30 00:52:22.649756 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:22.649719 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8982-207bf0"] Oct 30 00:52:23.443329 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:23.443290 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" event={"ID":"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721","Type":"ContainerStarted","Data":"2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e"} Oct 30 00:52:23.443547 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:23.443346 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" event={"ID":"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721","Type":"ContainerStarted","Data":"f0f2bec14bcfd20b14ee2c5e162f4cd1f7494d08aa007e8027fb53968415bb17"} Oct 30 00:52:23.443547 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:23.443334 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8982-207bf0" podUID="116e2909-5e63-418f-8e1e-276816f05036" containerName="test-curl-8982-207bf0" containerID="cri-o://27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195" gracePeriod=2 Oct 30 00:52:23.503138 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:23.503076 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" podStartSLOduration=2.503060058 podStartE2EDuration="2.503060058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:23.497489637 +0000 UTC m=+1082.069415858" watchObservedRunningTime="2026-10-30 00:52:23.503060058 +0000 UTC m=+1082.074986280" Oct 30 00:52:23.510433 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:52:23.510394 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8982-207bf0\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="116e2909-5e63-418f-8e1e-276816f05036" pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:23.523259 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:52:23.523218 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8982-207bf0\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="116e2909-5e63-418f-8e1e-276816f05036" pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:24.035663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:24.035617 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:24.042142 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:24.042114 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:24.479593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:24.475163 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:24.511428 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:24.510187 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:52:25.694562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:25.694531 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:25.756098 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:25.755988 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116e2909-5e63-418f-8e1e-276816f05036-kube-api-access-79mmz\" (UniqueName: \"kubernetes.io/projected/116e2909-5e63-418f-8e1e-276816f05036-kube-api-access-79mmz\") pod \"116e2909-5e63-418f-8e1e-276816f05036\" (UID: \"116e2909-5e63-418f-8e1e-276816f05036\") " Oct 30 00:52:25.759643 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:25.759608 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116e2909-5e63-418f-8e1e-276816f05036-kube-api-access-79mmz" pod "116e2909-5e63-418f-8e1e-276816f05036" (UID: "116e2909-5e63-418f-8e1e-276816f05036"). InnerVolumeSpecName "kube-api-access-79mmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:25.859173 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:25.859137 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79mmz\" (UniqueName: \"kubernetes.io/projected/116e2909-5e63-418f-8e1e-276816f05036-kube-api-access-79mmz\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:52:26.491661 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.491620 4927 generic.go:356] "Generic (PLEG): container finished" podID="116e2909-5e63-418f-8e1e-276816f05036" containerID="27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195" exitCode=137 Oct 30 00:52:26.491885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.491721 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:26.491885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.491713 4927 scope.go:122] "RemoveContainer" containerID="27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195" Oct 30 00:52:26.509979 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.509887 4927 scope.go:122] "RemoveContainer" containerID="27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195" Oct 30 00:52:26.513616 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:52:26.513538 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195\": container with ID starting with 27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195 not found: ID does not exist" containerID="27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195" Oct 30 00:52:26.513616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.513596 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195"} err="failed to get container status \"27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195\": rpc error: code = NotFound desc = could not find container \"27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195\": container with ID starting with 27a8a07a78bb7179a1b79f3a0711ca963bcd3e59e3e594eb7cb07fa645a21195 not found: ID does not exist" Oct 30 00:52:26.518063 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:52:26.518019 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8982-207bf0\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="116e2909-5e63-418f-8e1e-276816f05036" pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:26.756214 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.756125 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m"] Oct 30 00:52:26.756980 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.756961 4927 cpu_manager.go:414] "containerMap: removing container" podUID="116e2909-5e63-418f-8e1e-276816f05036" containerName="test-curl-8982-207bf0" Oct 30 00:52:26.757060 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.756981 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116e2909-5e63-418f-8e1e-276816f05036" containerName="test-curl-8982-207bf0" Oct 30 00:52:26.757124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.757114 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="116e2909-5e63-418f-8e1e-276816f05036" containerName="test-curl-8982-207bf0" Oct 30 00:52:26.762700 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.762677 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:26.778692 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:52:26.777554 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8982-207bf0\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="116e2909-5e63-418f-8e1e-276816f05036" pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:26.780080 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.780032 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m"] Oct 30 00:52:26.783809 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.783762 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1e708b2f\"" Oct 30 00:52:26.783941 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.783810 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1e708b2f-dockercfg-lklhd\"" Oct 30 00:52:26.829790 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.829754 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5"] Oct 30 00:52:26.832544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.832516 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f871728-6bba-4564-a833-0920e8beaef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m\" (UID: \"0f871728-6bba-4564-a833-0920e8beaef9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:26.832725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.832704 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ll5\" (UniqueName: \"kubernetes.io/projected/0f871728-6bba-4564-a833-0920e8beaef9-kube-api-access-p5ll5\") pod \"payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m\" (UID: \"0f871728-6bba-4564-a833-0920e8beaef9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:26.835110 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.835085 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:26.853933 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:26.853899 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5"] Oct 30 00:52:27.040077 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:52:26.972091 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8982-207bf0\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="116e2909-5e63-418f-8e1e-276816f05036" pod="opendatahub/test-curl-8982-207bf0" Oct 30 00:52:27.107594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.059267 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdfl\" (UniqueName: \"kubernetes.io/projected/8366bc0c-1598-4b5d-accc-06464a648122-kube-api-access-tkdfl\") pod \"payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5\" (UID: \"8366bc0c-1598-4b5d-accc-06464a648122\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:27.107594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.059443 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f871728-6bba-4564-a833-0920e8beaef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m\" (UID: \"0f871728-6bba-4564-a833-0920e8beaef9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:27.196494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.145822 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8366bc0c-1598-4b5d-accc-06464a648122-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5\" (UID: \"8366bc0c-1598-4b5d-accc-06464a648122\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:27.196494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.147606 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ll5\" (UniqueName: \"kubernetes.io/projected/0f871728-6bba-4564-a833-0920e8beaef9-kube-api-access-p5ll5\") pod \"payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m\" (UID: \"0f871728-6bba-4564-a833-0920e8beaef9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:27.196494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.148461 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f871728-6bba-4564-a833-0920e8beaef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m\" (UID: \"0f871728-6bba-4564-a833-0920e8beaef9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:27.254835 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.253040 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8366bc0c-1598-4b5d-accc-06464a648122-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5\" (UID: \"8366bc0c-1598-4b5d-accc-06464a648122\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:27.254835 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.253320 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdfl\" (UniqueName: \"kubernetes.io/projected/8366bc0c-1598-4b5d-accc-06464a648122-kube-api-access-tkdfl\") pod \"payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5\" (UID: \"8366bc0c-1598-4b5d-accc-06464a648122\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:27.254835 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.254781 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8366bc0c-1598-4b5d-accc-06464a648122-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5\" (UID: \"8366bc0c-1598-4b5d-accc-06464a648122\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:27.320243 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.320122 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ll5\" (UniqueName: \"kubernetes.io/projected/0f871728-6bba-4564-a833-0920e8beaef9-kube-api-access-p5ll5\") pod \"payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m\" (UID: \"0f871728-6bba-4564-a833-0920e8beaef9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:27.345601 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.345300 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdfl\" (UniqueName: \"kubernetes.io/projected/8366bc0c-1598-4b5d-accc-06464a648122-kube-api-access-tkdfl\") pod \"payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5\" (UID: \"8366bc0c-1598-4b5d-accc-06464a648122\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:27.380854 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.380805 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:27.478127 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.476564 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:27.649636 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.649589 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116e2909-5e63-418f-8e1e-276816f05036" path="/var/lib/kubelet/pods/116e2909-5e63-418f-8e1e-276816f05036/volumes" Oct 30 00:52:27.958796 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:27.957599 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m"] Oct 30 00:52:28.011695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:28.011292 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5"] Oct 30 00:52:28.740657 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:28.725525 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" event={"ID":"8366bc0c-1598-4b5d-accc-06464a648122","Type":"ContainerStarted","Data":"8925073539c4b7fbe602831a26dd745eb1b2180f40609434109a414e0636c5bb"} Oct 30 00:52:28.740957 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:28.740916 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" event={"ID":"8366bc0c-1598-4b5d-accc-06464a648122","Type":"ContainerStarted","Data":"ef24211cf1a1271d859a874ac5219aecd67aac41ed535bfa1ea916777433254d"} Oct 30 00:52:28.741453 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:28.741429 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:28.744884 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:28.744861 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" event={"ID":"0f871728-6bba-4564-a833-0920e8beaef9","Type":"ContainerStarted","Data":"4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719"} Oct 30 00:52:28.745059 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:28.745036 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" event={"ID":"0f871728-6bba-4564-a833-0920e8beaef9","Type":"ContainerStarted","Data":"15897d3cad5b7e4c7c616b24764a85091544978190b5de2a1a727ed17e712cee"} Oct 30 00:52:28.746054 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:28.746033 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:28.919996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:28.911144 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" podStartSLOduration=2.911120559 podStartE2EDuration="2.911120559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:28.795858539 +0000 UTC m=+1087.367784762" watchObservedRunningTime="2026-10-30 00:52:28.911120559 +0000 UTC m=+1087.483046781" Oct 30 00:52:28.919996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:28.911870 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" podStartSLOduration=2.911854579 podStartE2EDuration="2.911854579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:28.847434721 +0000 UTC m=+1087.419360944" watchObservedRunningTime="2026-10-30 00:52:28.911854579 +0000 UTC m=+1087.483780802" Oct 30 00:52:30.266898 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.266846 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:52:30.267412 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.267151 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.323640 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.321682 4927 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:30.323640 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.321753 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.492795 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.492738 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t"] Oct 30 00:52:30.653286 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.653228 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.706596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.693917 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:30.795805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.795766 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t"] Oct 30 00:52:30.796050 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.796036 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h"] Oct 30 00:52:30.831810 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:30.831767 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:31.081816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.081715 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:31.188594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.175539 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-74b07396-dockercfg-hbfmr\"" Oct 30 00:52:31.227643 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.225707 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-74b07396\"" Oct 30 00:52:31.294712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.268328 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09475bc1-6998-4664-99e5-e56ee4aa8f3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t\" (UID: \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:31.294712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.268542 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp44c\" (UniqueName: \"kubernetes.io/projected/09475bc1-6998-4664-99e5-e56ee4aa8f3b-kube-api-access-dp44c\") pod \"payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t\" (UID: \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:31.663182 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.663137 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp44c\" (UniqueName: \"kubernetes.io/projected/09475bc1-6998-4664-99e5-e56ee4aa8f3b-kube-api-access-dp44c\") pod \"payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t\" (UID: \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:31.721082 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.721036 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09475bc1-6998-4664-99e5-e56ee4aa8f3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t\" (UID: \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:31.776042 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.775989 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:52:31.801012 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.800967 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:52:31.952663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.872797 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:52:31.952663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.875737 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09475bc1-6998-4664-99e5-e56ee4aa8f3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t\" (UID: \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:31.952663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:31.921712 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.075602 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.066736 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szntg\" (UniqueName: \"kubernetes.io/projected/b36c2f94-365f-4e7e-89f4-2bcb24407d95-kube-api-access-szntg\") pod \"payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h\" (UID: \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\") " pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:32.106029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.083730 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36c2f94-365f-4e7e-89f4-2bcb24407d95-plugins-config-volume\") pod \"payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h\" (UID: \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\") " pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:32.238877 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.238776 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:52:32.432633 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.399503 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szntg\" (UniqueName: \"kubernetes.io/projected/b36c2f94-365f-4e7e-89f4-2bcb24407d95-kube-api-access-szntg\") pod \"payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h\" (UID: \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\") " pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:32.562951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.562847 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36c2f94-365f-4e7e-89f4-2bcb24407d95-plugins-config-volume\") pod \"payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h\" (UID: \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\") " pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:32.722596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.513141 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp44c\" (UniqueName: \"kubernetes.io/projected/09475bc1-6998-4664-99e5-e56ee4aa8f3b-kube-api-access-dp44c\") pod \"payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t\" (UID: \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:32.731694 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.511123 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:32.760394 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.629060 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36c2f94-365f-4e7e-89f4-2bcb24407d95-plugins-config-volume\") pod \"payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h\" (UID: \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\") " pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:32.795910 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.734059 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h"] Oct 30 00:52:32.871655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.763453 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:33.013595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.013071 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szntg\" (UniqueName: \"kubernetes.io/projected/b36c2f94-365f-4e7e-89f4-2bcb24407d95-kube-api-access-szntg\") pod \"payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h\" (UID: \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\") " pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:33.013595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.872690 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:52:33.309965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:32.819280 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:33.309965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.145936 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:33.309965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.179764 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:52:33.309965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.179835 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:52:33.309965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.179873 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:52:33.309965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.179913 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:52:33.309965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.179946 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:52:33.309965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.179974 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:52:33.309965 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.180000 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:52:33.672320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:33.672260 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:52:34.264658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.153183 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:52:34.264658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.226299 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:52:34.352606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.329953 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:52:34.373247 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.373214 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:34.604054 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.604019 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8"] Oct 30 00:52:34.614354 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.540469 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:34.618951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.618924 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:34.690948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.690903 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9"] Oct 30 00:52:34.704542 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.704252 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.716308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.716280 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:52:34.814079 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.813988 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8"] Oct 30 00:52:34.857860 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:52:34.846703 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:52:34.859117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.859041 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9"] Oct 30 00:52:34.939733 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.939695 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmcbx\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-kube-api-access-cmcbx\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:34.939873 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.939772 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.939938 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.939897 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.939998 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.939939 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.939998 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.939980 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:34.940106 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.940091 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:34.940221 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.940192 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:34.940301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.940238 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:34.940301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.940262 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qzc\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-kube-api-access-97qzc\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.940466 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.940444 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9960c4b-6496-450a-987f-95370d602e37-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:34.940558 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.940539 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.940887 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.940851 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:34.940975 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.940938 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.941056 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.941033 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.943958 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.943284 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.944240 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.944211 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:34.944315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.944286 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:34.944431 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:34.944411 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9960c4b-6496-450a-987f-95370d602e37-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.138315 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.138369 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.138441 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9960c4b-6496-450a-987f-95370d602e37-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.138497 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmcbx\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-kube-api-access-cmcbx\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.138549 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.138807 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.138841 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.138868 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.138934 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.139011 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.139038 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.139062 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qzc\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-kube-api-access-97qzc\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.139218 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9960c4b-6496-450a-987f-95370d602e37-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.139322 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.139423 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.139464 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.139509 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.155616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.139535 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.198950 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.166135 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.198950 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.166942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.198950 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.167206 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.198950 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.168731 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.198950 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.169130 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.198950 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.170076 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9960c4b-6496-450a-987f-95370d602e37-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.198950 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.170322 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.201009 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.200164 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.201009 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.200460 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.270647 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.263929 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.396743 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.376693 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t"] Oct 30 00:52:35.458622 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.447063 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.458622 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.456124 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.458622 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.457832 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.464748 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.464672 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.470327 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.468287 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qzc\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-kube-api-access-97qzc\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.470327 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.469145 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:35.470327 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.469757 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmcbx\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-kube-api-access-cmcbx\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.471995 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.471226 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9960c4b-6496-450a-987f-95370d602e37-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.595808 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.595777 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h"] Oct 30 00:52:35.598857 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:52:35.598820 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36c2f94_365f_4e7e_89f4_2bcb24407d95.slice/crio-d52a57f2339b24095268f828f5761a16e8e89e10eec7d9372e6103016a82e7b5 WatchSource:0}: Error finding container d52a57f2339b24095268f828f5761a16e8e89e10eec7d9372e6103016a82e7b5: Status 404 returned error can't find the container with id d52a57f2339b24095268f828f5761a16e8e89e10eec7d9372e6103016a82e7b5 Oct 30 00:52:35.615763 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.615732 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:35.640003 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.639955 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" event={"ID":"09475bc1-6998-4664-99e5-e56ee4aa8f3b","Type":"ContainerStarted","Data":"3449b97284564b8a90c1ed18d86f557533a3b2125a5e35cde1776bfba0c8fc3b"} Oct 30 00:52:35.640315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.640293 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:35.640438 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.640328 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" event={"ID":"09475bc1-6998-4664-99e5-e56ee4aa8f3b","Type":"ContainerStarted","Data":"f8a977a30bad2ad20dbe8ceb209df3db138bebb38182dae743410140e136aecf"} Oct 30 00:52:35.642054 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.642028 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" event={"ID":"b36c2f94-365f-4e7e-89f4-2bcb24407d95","Type":"ContainerStarted","Data":"d52a57f2339b24095268f828f5761a16e8e89e10eec7d9372e6103016a82e7b5"} Oct 30 00:52:35.705173 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:35.704894 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:36.050635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.048853 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" podStartSLOduration=6.048827681 podStartE2EDuration="6.048827681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:35.679759172 +0000 UTC m=+1094.251685395" watchObservedRunningTime="2026-10-30 00:52:36.048827681 +0000 UTC m=+1094.620753904" Oct 30 00:52:36.050635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.049297 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8"] Oct 30 00:52:36.064661 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.064601 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:36.064983 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.064957 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:36.065161 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.065134 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:36.092668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.092597 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9"] Oct 30 00:52:36.101989 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.101953 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:36.102077 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.102032 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:36.102077 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.102069 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:36.653563 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.653526 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" event={"ID":"4e5b9fe3-ac8b-4727-99ac-47a588050de8","Type":"ContainerStarted","Data":"191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06"} Oct 30 00:52:36.653563 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.653599 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" event={"ID":"4e5b9fe3-ac8b-4727-99ac-47a588050de8","Type":"ContainerStarted","Data":"169b380ceb23d62920a96d928707376a9872183e15fc63177efee1b1efe6a459"} Oct 30 00:52:36.655871 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.655843 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" event={"ID":"b36c2f94-365f-4e7e-89f4-2bcb24407d95","Type":"ContainerStarted","Data":"1364c8f077555e28479cdedb5e3a887332ad184e1be40e6c8d46e1f6dd20cedb"} Oct 30 00:52:36.656007 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.655894 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:36.657684 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.657655 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" event={"ID":"d9960c4b-6496-450a-987f-95370d602e37","Type":"ContainerStarted","Data":"c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a"} Oct 30 00:52:36.657816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.657700 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" event={"ID":"d9960c4b-6496-450a-987f-95370d602e37","Type":"ContainerStarted","Data":"684bed5899ce2a5ff0cbf07cfc8cce328360472e63ad8a6f8ce24bbb8ee120f2"} Oct 30 00:52:36.703659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.703601 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" podStartSLOduration=3.703584179 podStartE2EDuration="3.703584179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:36.696820857 +0000 UTC m=+1095.268747082" watchObservedRunningTime="2026-10-30 00:52:36.703584179 +0000 UTC m=+1095.275510392" Oct 30 00:52:36.709552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.709520 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:36.741050 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.740994 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" podStartSLOduration=3.740978971 podStartE2EDuration="3.740978971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:36.738774468 +0000 UTC m=+1095.310700689" watchObservedRunningTime="2026-10-30 00:52:36.740978971 +0000 UTC m=+1095.312905194" Oct 30 00:52:36.784325 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:36.784261 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" podStartSLOduration=6.784241475 podStartE2EDuration="6.784241475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:36.776504422 +0000 UTC m=+1095.348430659" watchObservedRunningTime="2026-10-30 00:52:36.784241475 +0000 UTC m=+1095.356167701" Oct 30 00:52:37.642599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:37.640757 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:52:37.642599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:37.640831 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:52:37.643247 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:37.643181 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:37.720594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:37.720307 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:52:37.720594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:37.720375 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 00:52:38.632598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:38.632414 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:52:38.632598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:38.632478 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:52:38.721584 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:38.721525 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:52:38.722178 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:38.722140 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 00:52:39.678593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:39.652974 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:52:39.682595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:39.678894 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:52:39.748212 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:39.711618 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:52:39.752609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:39.748888 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 00:52:39.976651 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:39.975818 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:52:40.617605 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:40.617113 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:52:40.617605 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:40.617192 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:52:40.719184 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:40.718705 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:52:40.719184 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:40.718769 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 00:52:41.662765 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:41.662724 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:52:41.663178 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:41.662791 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:52:41.710092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:41.710049 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:52:41.710412 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:41.710381 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 00:52:42.625116 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:42.625074 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:52:42.625431 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:42.625396 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:52:42.713062 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:42.713018 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:52:42.713718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:42.713681 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 00:52:42.770437 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:42.770402 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:52:43.617652 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:43.617553 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 00:52:43.617825 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:43.617686 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 00:52:43.709681 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:43.709637 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 00:52:43.709851 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:43.709711 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 00:52:44.622279 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:44.622242 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:44.622817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:44.622795 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:44.623624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:44.623600 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:52:44.713931 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:44.713899 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:44.714219 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:44.714200 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:44.714909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:44.714888 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 00:52:46.662212 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:46.662178 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:52:47.670962 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:47.670926 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:52:52.302212 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.302174 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx"] Oct 30 00:52:52.321550 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.321512 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.334406 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.334368 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx"] Oct 30 00:52:52.336970 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.336937 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c4e6c558-serving-cert\"" Oct 30 00:52:52.486908 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.486869 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nx4\" (UniqueName: \"kubernetes.io/projected/60fa29a9-51cb-4c79-a1d4-a6a802ede342-kube-api-access-h8nx4\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.487117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.487042 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-maas-api-tls\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.487291 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.487261 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fa29a9-51cb-4c79-a1d4-a6a802ede342-openshift-service-ca\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.487420 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.487303 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-metrics-certs\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.603015 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.602919 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fa29a9-51cb-4c79-a1d4-a6a802ede342-openshift-service-ca\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.622738 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.622692 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-metrics-certs\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.623206 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.623182 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nx4\" (UniqueName: \"kubernetes.io/projected/60fa29a9-51cb-4c79-a1d4-a6a802ede342-kube-api-access-h8nx4\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.639820 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.639775 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-maas-api-tls\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.642624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.606733 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fa29a9-51cb-4c79-a1d4-a6a802ede342-openshift-service-ca\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.671979 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.671945 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nx4\" (UniqueName: \"kubernetes.io/projected/60fa29a9-51cb-4c79-a1d4-a6a802ede342-kube-api-access-h8nx4\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.688224 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.688191 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-metrics-certs\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.693061 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.692208 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-maas-api-tls\") pod \"maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:52.956690 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:52.956656 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:53.286925 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:53.286892 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx"] Oct 30 00:52:53.287368 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:52:53.287334 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fa29a9_51cb_4c79_a1d4_a6a802ede342.slice/crio-7d7fe3b4953aad8ba7f8878d6d33d53965c4dbf402396feb87e290aaf90b6e54 WatchSource:0}: Error finding container 7d7fe3b4953aad8ba7f8878d6d33d53965c4dbf402396feb87e290aaf90b6e54: Status 404 returned error can't find the container with id 7d7fe3b4953aad8ba7f8878d6d33d53965c4dbf402396feb87e290aaf90b6e54 Oct 30 00:52:54.022842 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:54.022803 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" event={"ID":"60fa29a9-51cb-4c79-a1d4-a6a802ede342","Type":"ContainerStarted","Data":"bb101d9016118cd99c7ddf552a82afcc828b3fc42f5fa95acbf478ea3bc78aed"} Oct 30 00:52:54.023278 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:54.022871 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" event={"ID":"60fa29a9-51cb-4c79-a1d4-a6a802ede342","Type":"ContainerStarted","Data":"7d7fe3b4953aad8ba7f8878d6d33d53965c4dbf402396feb87e290aaf90b6e54"} Oct 30 00:52:54.023278 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:54.022926 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:52:54.064215 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:52:54.064154 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" podStartSLOduration=1.749535967 podStartE2EDuration="2.064137393s" totalImagesPullingTime="314.601426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:54.062353071 +0000 UTC m=+1112.634279295" watchObservedRunningTime="2026-10-30 00:52:54.064137393 +0000 UTC m=+1112.636063629" Oct 30 00:53:00.036011 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:00.035980 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:53:02.601654 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:02.592690 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:02.692764 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:02.692696 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:03.268092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:03.268035 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:03.609647 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:03.601111 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:03.617663 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:03.616501 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:53:03.761599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:03.746496 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:53:03.778598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:03.749701 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:53:03.799595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:03.783872 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:53:04.043596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:03.941704 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:53:04.332655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:04.329742 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.379884 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:04.368862 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:53:04.637660 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:04.609160 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 00:53:04.641721 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:04.641674 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:04.794599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:04.792176 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:53:04.921658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:04.880633 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:53:04.921658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:04.882534 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:53:05.066503 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:05.051236 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:53:05.066503 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:05.051869 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:53:05.265661 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:05.263270 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.365593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:05.349680 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.577457 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:05.575646 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:53:05.652787 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:05.652730 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:05.819212 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:05.819151 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:06.112609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:06.110299 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:53:06.178967 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:06.178900 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:06.379715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:06.373770 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:53:06.857229 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:06.857115 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:53:06.925598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:06.900329 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:53:07.114822 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:07.114697 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:07.189217 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:07.184117 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:07.388598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:07.387045 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:53:07.388598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:07.387725 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:53:07.762994 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:07.762885 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:07.876598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:07.875284 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:08.012597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:08.008731 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:08.188433 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:08.188376 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:08.190104 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:08.190052 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:08.409613 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:08.391588 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:08.652662 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:08.621100 4927 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.054s" Oct 30 00:53:08.990468 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:08.990365 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:53:09.109823 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:09.109782 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:53:09.211602 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:09.206118 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.431749 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:09.209759 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:09.599595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:09.520163 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:53:10.016530 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:10.016479 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Oct 30 00:53:10.035731 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:10.034768 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk" podUID="c19cce81-4a48-4b54-b2ac-29ef57f9395c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:10.036209 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:10.034825 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:53:10.169159 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:10.153461 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:10.444847 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:10.444802 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:53:10.528754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:10.528706 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:10.630180 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:10.630130 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Oct 30 00:53:10.656972 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:10.656921 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:53:11.139179 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:11.139125 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:11.209174 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:11.209141 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:53:11.455652 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:11.454809 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:53:11.455854 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:11.455672 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:53:11.537179 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:11.536682 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:53:11.730862 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:11.730740 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:53:12.138882 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:12.124709 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:53:12.471712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:12.457557 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:12.504436 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:12.504392 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:53:12.540201 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:12.540150 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:12.728704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:12.714288 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:53:17.045027 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.044990 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7"] Oct 30 00:53:17.056974 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.056907 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:17.074984 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.074950 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7"] Oct 30 00:53:17.076263 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.076187 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-ea2547\"" Oct 30 00:53:17.076263 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.076253 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-ea2547-dockercfg-jbmvb\"" Oct 30 00:53:17.512604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.507842 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7\" (UID: \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:17.512604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.507950 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfp66\" (UniqueName: \"kubernetes.io/projected/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-kube-api-access-hfp66\") pod \"payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7\" (UID: \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:17.607708 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.607671 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b"] Oct 30 00:53:17.611615 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.610841 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7\" (UID: \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:17.611615 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.610935 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfp66\" (UniqueName: \"kubernetes.io/projected/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-kube-api-access-hfp66\") pod \"payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7\" (UID: \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:17.611989 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.611963 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7\" (UID: \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:17.613887 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.613330 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:17.618650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.616241 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b"] Oct 30 00:53:17.640909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.640875 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfp66\" (UniqueName: \"kubernetes.io/projected/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-kube-api-access-hfp66\") pod \"payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7\" (UID: \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:17.678660 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.678549 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:17.712305 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.712270 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2pz7\" (UniqueName: \"kubernetes.io/projected/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-kube-api-access-n2pz7\") pod \"payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b\" (UID: \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:17.712755 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.712731 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b\" (UID: \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:17.819592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.816826 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b\" (UID: \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:17.819592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.817920 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b\" (UID: \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:17.819592 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.819477 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2pz7\" (UniqueName: \"kubernetes.io/projected/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-kube-api-access-n2pz7\") pod \"payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b\" (UID: \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:17.845773 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.845730 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2pz7\" (UniqueName: \"kubernetes.io/projected/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-kube-api-access-n2pz7\") pod \"payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b\" (UID: \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:17.937552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:17.937511 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:18.057733 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.053226 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7"] Oct 30 00:53:18.432185 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:53:18.431412 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5725f313_f8ee_40cd_9ef8_f17e18d8ab60.slice/crio-61f0bcc933dbbf47e3a6bc788b9e1be937975d15042a0e5490138d32038a5cdf WatchSource:0}: Error finding container 61f0bcc933dbbf47e3a6bc788b9e1be937975d15042a0e5490138d32038a5cdf: Status 404 returned error can't find the container with id 61f0bcc933dbbf47e3a6bc788b9e1be937975d15042a0e5490138d32038a5cdf Oct 30 00:53:18.434817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.434421 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b"] Oct 30 00:53:18.595878 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.595783 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" event={"ID":"5725f313-f8ee-40cd-9ef8-f17e18d8ab60","Type":"ContainerStarted","Data":"163076a131f8a6395fc1e891f2b01cc5f9666547e61e0f2ab21eef5c5d2a36f2"} Oct 30 00:53:18.595878 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.595846 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" event={"ID":"5725f313-f8ee-40cd-9ef8-f17e18d8ab60","Type":"ContainerStarted","Data":"61f0bcc933dbbf47e3a6bc788b9e1be937975d15042a0e5490138d32038a5cdf"} Oct 30 00:53:18.596079 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.595900 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:18.597812 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.597777 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" event={"ID":"e22f4c73-42a9-41b9-8f86-2dd9b4d41779","Type":"ContainerStarted","Data":"d6f0d3583acd48fa1d588fa8b7dfc7aabc890847815dbf2c887e40e26c483107"} Oct 30 00:53:18.597972 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.597829 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" event={"ID":"e22f4c73-42a9-41b9-8f86-2dd9b4d41779","Type":"ContainerStarted","Data":"9711991f5ef3e65a8b21d887a0271dc12a9cfdcc56f66feb4585e36024765974"} Oct 30 00:53:18.597972 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.597873 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:18.639031 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.638960 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" podStartSLOduration=1.63894199 podStartE2EDuration="1.63894199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:18.633937113 +0000 UTC m=+1137.205863335" watchObservedRunningTime="2026-10-30 00:53:18.63894199 +0000 UTC m=+1137.210868209" Oct 30 00:53:18.673736 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:18.673672 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" podStartSLOduration=1.673656829 podStartE2EDuration="1.673656829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:18.67148942 +0000 UTC m=+1137.243415644" watchObservedRunningTime="2026-10-30 00:53:18.673656829 +0000 UTC m=+1137.245583050" Oct 30 00:53:20.756565 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:20.756526 4927 generic.go:356] "Generic (PLEG): container finished" podID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerID="2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e" exitCode=0 Oct 30 00:53:20.757160 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:20.756607 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" event={"ID":"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721","Type":"ContainerDied","Data":"2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e"} Oct 30 00:53:20.757254 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:20.757236 4927 scope.go:122] "RemoveContainer" containerID="2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e" Oct 30 00:53:20.759460 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:20.759423 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:20.759558 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:20.759526 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:20.759645 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:20.759595 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:21.840828 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:21.840776 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" event={"ID":"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721","Type":"ContainerStarted","Data":"4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166"} Oct 30 00:53:22.051594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:22.047782 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:53:22.109921 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:22.109811 4927 patch_prober.go:29] interesting pod/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:53:22.110090 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:22.109918 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:53:23.036703 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:23.036665 4927 patch_prober.go:29] interesting pod/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:53:23.037129 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:23.036738 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:53:24.036228 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:24.036188 4927 patch_prober.go:29] interesting pod/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:53:24.036414 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:24.036267 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:53:24.490598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:24.477689 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:53:25.035746 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:25.035710 4927 patch_prober.go:29] interesting pod/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:53:25.035934 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:25.035771 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:53:26.036252 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:26.036216 4927 patch_prober.go:29] interesting pod/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:53:26.036704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:26.036274 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:53:26.835330 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:26.835284 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5"] Oct 30 00:53:26.871842 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:26.871802 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m"] Oct 30 00:53:27.035801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:27.035767 4927 patch_prober.go:29] interesting pod/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:53:27.036000 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:27.035831 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 00:53:28.041033 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:28.041001 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:53:28.042155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:28.042133 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:53:29.747983 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:29.747950 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 00:53:29.748528 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:29.748375 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 00:53:31.836530 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:31.836483 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" podUID="8366bc0c-1598-4b5d-accc-06464a648122" containerName="payload-processing" containerID="cri-o://8925073539c4b7fbe602831a26dd745eb1b2180f40609434109a414e0636c5bb" gracePeriod=25 Oct 30 00:53:31.872997 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:31.872932 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" podUID="0f871728-6bba-4564-a833-0920e8beaef9" containerName="payload-pre-processing" containerID="cri-o://4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719" gracePeriod=25 Oct 30 00:53:32.019411 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.019365 4927 generic.go:356] "Generic (PLEG): container finished" podID="8366bc0c-1598-4b5d-accc-06464a648122" containerID="8925073539c4b7fbe602831a26dd745eb1b2180f40609434109a414e0636c5bb" exitCode=0 Oct 30 00:53:32.019767 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.019734 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" event={"ID":"8366bc0c-1598-4b5d-accc-06464a648122","Type":"ContainerDied","Data":"8925073539c4b7fbe602831a26dd745eb1b2180f40609434109a414e0636c5bb"} Oct 30 00:53:32.261238 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.256321 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:53:32.284773 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.284676 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8366bc0c-1598-4b5d-accc-06464a648122-kube-api-access-tkdfl\" (UniqueName: \"kubernetes.io/projected/8366bc0c-1598-4b5d-accc-06464a648122-kube-api-access-tkdfl\") pod \"8366bc0c-1598-4b5d-accc-06464a648122\" (UID: \"8366bc0c-1598-4b5d-accc-06464a648122\") " Oct 30 00:53:32.284987 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.284890 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8366bc0c-1598-4b5d-accc-06464a648122-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8366bc0c-1598-4b5d-accc-06464a648122-plugins-config-volume\") pod \"8366bc0c-1598-4b5d-accc-06464a648122\" (UID: \"8366bc0c-1598-4b5d-accc-06464a648122\") " Oct 30 00:53:32.285270 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.285240 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8366bc0c-1598-4b5d-accc-06464a648122-plugins-config-volume" pod "8366bc0c-1598-4b5d-accc-06464a648122" (UID: "8366bc0c-1598-4b5d-accc-06464a648122"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:32.285470 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.285456 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8366bc0c-1598-4b5d-accc-06464a648122-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:32.287200 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.287173 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8366bc0c-1598-4b5d-accc-06464a648122-kube-api-access-tkdfl" pod "8366bc0c-1598-4b5d-accc-06464a648122" (UID: "8366bc0c-1598-4b5d-accc-06464a648122"). InnerVolumeSpecName "kube-api-access-tkdfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:32.319593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.302660 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:53:32.386495 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.386445 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f871728-6bba-4564-a833-0920e8beaef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f871728-6bba-4564-a833-0920e8beaef9-plugins-config-volume\") pod \"0f871728-6bba-4564-a833-0920e8beaef9\" (UID: \"0f871728-6bba-4564-a833-0920e8beaef9\") " Oct 30 00:53:32.386691 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.386673 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f871728-6bba-4564-a833-0920e8beaef9-kube-api-access-p5ll5\" (UniqueName: \"kubernetes.io/projected/0f871728-6bba-4564-a833-0920e8beaef9-kube-api-access-p5ll5\") pod \"0f871728-6bba-4564-a833-0920e8beaef9\" (UID: \"0f871728-6bba-4564-a833-0920e8beaef9\") " Oct 30 00:53:32.386997 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.386957 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f871728-6bba-4564-a833-0920e8beaef9-plugins-config-volume" pod "0f871728-6bba-4564-a833-0920e8beaef9" (UID: "0f871728-6bba-4564-a833-0920e8beaef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:32.387845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.387816 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdfl\" (UniqueName: \"kubernetes.io/projected/8366bc0c-1598-4b5d-accc-06464a648122-kube-api-access-tkdfl\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:32.387988 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.387850 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f871728-6bba-4564-a833-0920e8beaef9-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:32.389466 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.389427 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f871728-6bba-4564-a833-0920e8beaef9-kube-api-access-p5ll5" pod "0f871728-6bba-4564-a833-0920e8beaef9" (UID: "0f871728-6bba-4564-a833-0920e8beaef9"). InnerVolumeSpecName "kube-api-access-p5ll5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:32.489492 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:32.489449 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ll5\" (UniqueName: \"kubernetes.io/projected/0f871728-6bba-4564-a833-0920e8beaef9-kube-api-access-p5ll5\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:33.030193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.030148 4927 generic.go:356] "Generic (PLEG): container finished" podID="0f871728-6bba-4564-a833-0920e8beaef9" containerID="4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719" exitCode=0 Oct 30 00:53:33.030642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.030221 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" Oct 30 00:53:33.030642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.030233 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" event={"ID":"0f871728-6bba-4564-a833-0920e8beaef9","Type":"ContainerDied","Data":"4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719"} Oct 30 00:53:33.030642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.030319 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m" event={"ID":"0f871728-6bba-4564-a833-0920e8beaef9","Type":"ContainerDied","Data":"15897d3cad5b7e4c7c616b24764a85091544978190b5de2a1a727ed17e712cee"} Oct 30 00:53:33.030642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.030352 4927 scope.go:122] "RemoveContainer" containerID="4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719" Oct 30 00:53:33.032032 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.032006 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" event={"ID":"8366bc0c-1598-4b5d-accc-06464a648122","Type":"ContainerDied","Data":"ef24211cf1a1271d859a874ac5219aecd67aac41ed535bfa1ea916777433254d"} Oct 30 00:53:33.032274 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.032253 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5" Oct 30 00:53:33.042718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.042688 4927 scope.go:122] "RemoveContainer" containerID="4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719" Oct 30 00:53:33.043105 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:33.043075 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719\": container with ID starting with 4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719 not found: ID does not exist" containerID="4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719" Oct 30 00:53:33.043200 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.043114 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719"} err="failed to get container status \"4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719\": rpc error: code = NotFound desc = could not find container \"4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719\": container with ID starting with 4bbae88bfb781afb76b9681777e9c2c7cbe7478a4736fa0d6f837695ae782719 not found: ID does not exist" Oct 30 00:53:33.043200 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.043134 4927 scope.go:122] "RemoveContainer" containerID="8925073539c4b7fbe602831a26dd745eb1b2180f40609434109a414e0636c5bb" Oct 30 00:53:33.082224 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.082180 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m"] Oct 30 00:53:33.093307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.093272 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e708b2f-666c96b4d7-4tm2m"] Oct 30 00:53:33.132845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.130364 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5"] Oct 30 00:53:33.141282 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.141242 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e708b2f-55698bbdc8-5gmp5"] Oct 30 00:53:33.371733 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.370885 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h"] Oct 30 00:53:33.447651 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.443488 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t"] Oct 30 00:53:33.826802 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.826712 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f871728-6bba-4564-a833-0920e8beaef9" path="/var/lib/kubelet/pods/0f871728-6bba-4564-a833-0920e8beaef9/volumes" Oct 30 00:53:33.827957 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:33.827931 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8366bc0c-1598-4b5d-accc-06464a648122" path="/var/lib/kubelet/pods/8366bc0c-1598-4b5d-accc-06464a648122/volumes" Oct 30 00:53:34.077221 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.071594 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h"] Oct 30 00:53:34.163826 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.163795 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0f871728-6bba-4564-a833-0920e8beaef9" containerName="payload-pre-processing" Oct 30 00:53:34.164196 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.164174 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f871728-6bba-4564-a833-0920e8beaef9" containerName="payload-pre-processing" Oct 30 00:53:34.164935 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.164912 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8366bc0c-1598-4b5d-accc-06464a648122" containerName="payload-processing" Oct 30 00:53:34.165069 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.165056 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8366bc0c-1598-4b5d-accc-06464a648122" containerName="payload-processing" Oct 30 00:53:34.165386 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.165369 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f871728-6bba-4564-a833-0920e8beaef9" containerName="payload-pre-processing" Oct 30 00:53:34.165505 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.165493 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8366bc0c-1598-4b5d-accc-06464a648122" containerName="payload-processing" Oct 30 00:53:34.186631 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.181917 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h"] Oct 30 00:53:34.186631 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.182062 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.405596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.391053 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.405596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.391148 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.405596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.391195 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.405596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.391226 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.405596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.391425 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.405596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.391454 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxdr\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-kube-api-access-5wxdr\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.405596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.391502 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.405596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.391555 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.405596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.391626 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.510648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.507305 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.510648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.507361 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxdr\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-kube-api-access-5wxdr\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.510648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.507413 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.510648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.507464 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.510648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.507510 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.510648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.508001 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.510648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.508134 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.510648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.508181 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.510648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.508212 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.535598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.522966 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.535598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.524281 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.535598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.524587 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.535598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.524847 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.535598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.525125 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.549625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.546199 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.549625 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.546408 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.574811 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.558262 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.637593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.617552 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxdr\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-kube-api-access-5wxdr\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:34.673657 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:34.664589 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:53:34.811311 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:34.811270 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:35.496272 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:35.496227 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h"] Oct 30 00:53:35.533844 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:35.532670 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:35.533844 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:35.532772 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:35.533844 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:35.532816 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:36.355658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.353743 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" event={"ID":"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9","Type":"ContainerStarted","Data":"aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1"} Oct 30 00:53:36.355658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.353842 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" event={"ID":"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9","Type":"ContainerStarted","Data":"b9a915feffb1338b9c005eb171171abd3d03ea6e03111fbb846b7fd08999c7bb"} Oct 30 00:53:36.494603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.489737 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podStartSLOduration=3.489714719 podStartE2EDuration="3.489714719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:36.475161672 +0000 UTC m=+1155.047087895" watchObservedRunningTime="2026-10-30 00:53:36.489714719 +0000 UTC m=+1155.061640941" Oct 30 00:53:36.618672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.616627 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm"] Oct 30 00:53:36.833417 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.833376 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm"] Oct 30 00:53:36.833808 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.833791 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:36.898750 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.898716 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:36.923287 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.923187 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61512ccc-12bf-4206-9163-d11212cdba6d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:36.923856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.923817 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.014524 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:36.981068 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-ea2547-serving-cert\"" Oct 30 00:53:37.014896 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.014868 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.015257 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.015234 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb67v\" (UniqueName: \"kubernetes.io/projected/61512ccc-12bf-4206-9163-d11212cdba6d-kube-api-access-gb67v\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.121609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.119166 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.121609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.119254 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.121609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.119481 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb67v\" (UniqueName: \"kubernetes.io/projected/61512ccc-12bf-4206-9163-d11212cdba6d-kube-api-access-gb67v\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.121609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.119525 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61512ccc-12bf-4206-9163-d11212cdba6d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.121609 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.120671 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61512ccc-12bf-4206-9163-d11212cdba6d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.121609 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:37.120798 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-serving-cert: secret "maas-api-e2e-shared-a-w6-ea2547-serving-cert" not found Oct 30 00:53:37.121609 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:37.120860 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls podName:61512ccc-12bf-4206-9163-d11212cdba6d nodeName:}" failed. No retries permitted until 2026-10-30 00:53:37.620838238 +0000 UTC m=+1156.192764455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls") pod "maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" (UID: "61512ccc-12bf-4206-9163-d11212cdba6d") : secret "maas-api-e2e-shared-a-w6-ea2547-serving-cert" not found Oct 30 00:53:37.141598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.135312 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.239772 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.222197 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb67v\" (UniqueName: \"kubernetes.io/projected/61512ccc-12bf-4206-9163-d11212cdba6d-kube-api-access-gb67v\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.577669 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.413287 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.717608 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.685817 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:37.775310 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:37.745804 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-serving-cert: secret "maas-api-e2e-shared-a-w6-ea2547-serving-cert" not found Oct 30 00:53:37.775643 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:37.775623 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls podName:61512ccc-12bf-4206-9163-d11212cdba6d nodeName:}" failed. No retries permitted until 2026-10-30 00:53:38.77559148 +0000 UTC m=+1157.347517689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls") pod "maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" (UID: "61512ccc-12bf-4206-9163-d11212cdba6d") : secret "maas-api-e2e-shared-a-w6-ea2547-serving-cert" not found Oct 30 00:53:37.776154 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.746987 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 00:53:37.811597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.802104 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.945453 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.945382 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:37.949111 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:37.949067 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:38.120169 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.120121 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.281125 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.250917 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.393713 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.377462 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" podUID="b36c2f94-365f-4e7e-89f4-2bcb24407d95" containerName="payload-processing" containerID="cri-o://1364c8f077555e28479cdedb5e3a887332ad184e1be40e6c8d46e1f6dd20cedb" gracePeriod=25 Oct 30 00:53:38.441219 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.432350 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:53:38.516865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.515912 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" podUID="09475bc1-6998-4664-99e5-e56ee4aa8f3b" containerName="payload-pre-processing" containerID="cri-o://3449b97284564b8a90c1ed18d86f557533a3b2125a5e35cde1776bfba0c8fc3b" gracePeriod=25 Oct 30 00:53:38.773600 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:38.755751 4927 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.137s" Oct 30 00:53:38.773600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.755870 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:53:38.815599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.789650 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.894597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.877756 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:38.922596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.894909 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:38.922909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.891247 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:53:38.934598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.839445 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:38.956612 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:38.952715 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:53:39.113187 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.113140 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:39.130767 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.129967 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:39.130767 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.130340 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:53:39.351366 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.351320 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 00:53:39.363916 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.363867 4927 generic.go:356] "Generic (PLEG): container finished" podID="09475bc1-6998-4664-99e5-e56ee4aa8f3b" containerID="3449b97284564b8a90c1ed18d86f557533a3b2125a5e35cde1776bfba0c8fc3b" exitCode=0 Oct 30 00:53:39.365212 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.365183 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" event={"ID":"09475bc1-6998-4664-99e5-e56ee4aa8f3b","Type":"ContainerDied","Data":"3449b97284564b8a90c1ed18d86f557533a3b2125a5e35cde1776bfba0c8fc3b"} Oct 30 00:53:39.456361 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.440755 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:39.807680 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.805349 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:39.841814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.819361 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:39.841814 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.819427 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:39.882495 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.882456 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:39.953347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:39.933868 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.272885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:40.171483 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:53:40.704599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:40.276156 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:53:40.704599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:40.550641 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:40.973741 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:40.958804 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:40.973741 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:40.958897 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:40.973741 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:40.962456 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 00:53:40.973741 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:40.965305 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:53:41.310607 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:41.306053 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 00:53:41.900597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:41.898364 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:41.900597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:41.900503 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:53:41.911616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:41.906889 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:41.911616 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:41.906949 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:41.989533 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:41.989479 4927 generic.go:356] "Generic (PLEG): container finished" podID="b36c2f94-365f-4e7e-89f4-2bcb24407d95" containerID="1364c8f077555e28479cdedb5e3a887332ad184e1be40e6c8d46e1f6dd20cedb" exitCode=0 Oct 30 00:53:42.048496 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.043491 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" event={"ID":"b36c2f94-365f-4e7e-89f4-2bcb24407d95","Type":"ContainerDied","Data":"1364c8f077555e28479cdedb5e3a887332ad184e1be40e6c8d46e1f6dd20cedb"} Oct 30 00:53:42.062437 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:42.062319 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36c2f94_365f_4e7e_89f4_2bcb24407d95.slice/crio-conmon-1364c8f077555e28479cdedb5e3a887332ad184e1be40e6c8d46e1f6dd20cedb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09475bc1_6998_4664_99e5_e56ee4aa8f3b.slice/crio-conmon-3449b97284564b8a90c1ed18d86f557533a3b2125a5e35cde1776bfba0c8fc3b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09475bc1_6998_4664_99e5_e56ee4aa8f3b.slice/crio-3449b97284564b8a90c1ed18d86f557533a3b2125a5e35cde1776bfba0c8fc3b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 00:53:42.251850 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.251751 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 00:53:42.331605 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.317550 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:42.535833 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.535723 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:42.660979 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.660922 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:53:42.661194 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.661177 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2"] Oct 30 00:53:42.664769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.664370 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" containerID="cri-o://4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166" gracePeriod=30 Oct 30 00:53:42.751635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.747693 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn"] Oct 30 00:53:42.754293 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.754265 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:53:42.755761 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.755729 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn"] Oct 30 00:53:42.755898 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.755866 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.758020 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.757995 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:53:42.810012 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.809920 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-573837e3-openshift-default-dockercfg-bvdcc\"" Oct 30 00:53:42.812818 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.812782 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:42.812977 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.812836 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:42.832973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.832931 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09475bc1-6998-4664-99e5-e56ee4aa8f3b-kube-api-access-dp44c\" (UniqueName: \"kubernetes.io/projected/09475bc1-6998-4664-99e5-e56ee4aa8f3b-kube-api-access-dp44c\") pod \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\" (UID: \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\") " Oct 30 00:53:42.833199 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.833176 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09475bc1-6998-4664-99e5-e56ee4aa8f3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09475bc1-6998-4664-99e5-e56ee4aa8f3b-plugins-config-volume\") pod \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\" (UID: \"09475bc1-6998-4664-99e5-e56ee4aa8f3b\") " Oct 30 00:53:42.833343 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.833215 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36c2f94-365f-4e7e-89f4-2bcb24407d95-kube-api-access-szntg\" (UniqueName: \"kubernetes.io/projected/b36c2f94-365f-4e7e-89f4-2bcb24407d95-kube-api-access-szntg\") pod \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\" (UID: \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\") " Oct 30 00:53:42.833421 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.833404 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36c2f94-365f-4e7e-89f4-2bcb24407d95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36c2f94-365f-4e7e-89f4-2bcb24407d95-plugins-config-volume\") pod \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\" (UID: \"b36c2f94-365f-4e7e-89f4-2bcb24407d95\") " Oct 30 00:53:42.833650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.833627 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-socket\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.833748 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.833732 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-token\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.834058 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cmmg\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-kube-api-access-6cmmg\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.834106 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36c2f94-365f-4e7e-89f4-2bcb24407d95-plugins-config-volume" pod "b36c2f94-365f-4e7e-89f4-2bcb24407d95" (UID: "b36c2f94-365f-4e7e-89f4-2bcb24407d95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.834171 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-envoy\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.834202 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-certs\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.834293 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-credential-socket\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.834748 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istiod-ca-cert\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.834785 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-data\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.834902 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-podinfo\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.835146 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09475bc1-6998-4664-99e5-e56ee4aa8f3b-plugins-config-volume" pod "09475bc1-6998-4664-99e5-e56ee4aa8f3b" (UID: "09475bc1-6998-4664-99e5-e56ee4aa8f3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:42.835392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.835190 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b36c2f94-365f-4e7e-89f4-2bcb24407d95-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:42.836075 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.835689 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09475bc1-6998-4664-99e5-e56ee4aa8f3b-kube-api-access-dp44c" pod "09475bc1-6998-4664-99e5-e56ee4aa8f3b" (UID: "09475bc1-6998-4664-99e5-e56ee4aa8f3b"). InnerVolumeSpecName "kube-api-access-dp44c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:42.836316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.836293 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36c2f94-365f-4e7e-89f4-2bcb24407d95-kube-api-access-szntg" pod "b36c2f94-365f-4e7e-89f4-2bcb24407d95" (UID: "b36c2f94-365f-4e7e-89f4-2bcb24407d95"). InnerVolumeSpecName "kube-api-access-szntg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:42.862894 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:53:42.862863 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61512ccc_12bf_4206_9163_d11212cdba6d.slice/crio-c4ce26da8ee4c05661ff1eb98f54f3498b3d3452b5fce560780d5981db35019b WatchSource:0}: Error finding container c4ce26da8ee4c05661ff1eb98f54f3498b3d3452b5fce560780d5981db35019b: Status 404 returned error can't find the container with id c4ce26da8ee4c05661ff1eb98f54f3498b3d3452b5fce560780d5981db35019b Oct 30 00:53:42.865188 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.865156 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm"] Oct 30 00:53:42.936454 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.936401 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-podinfo\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.936454 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.936457 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-socket\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.936752 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.936553 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-token\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.936822 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.936799 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cmmg\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-kube-api-access-6cmmg\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.936889 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.936871 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-envoy\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.936943 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.936901 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-certs\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.936992 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.936979 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-credential-socket\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.937106 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.937091 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istiod-ca-cert\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.937159 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.937119 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-data\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.937253 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.937238 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09475bc1-6998-4664-99e5-e56ee4aa8f3b-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:42.937301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.937259 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szntg\" (UniqueName: \"kubernetes.io/projected/b36c2f94-365f-4e7e-89f4-2bcb24407d95-kube-api-access-szntg\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:42.937301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.937277 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp44c\" (UniqueName: \"kubernetes.io/projected/09475bc1-6998-4664-99e5-e56ee4aa8f3b-kube-api-access-dp44c\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:42.937301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.937276 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-socket\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.937565 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.937538 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-certs\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.937565 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.937560 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-credential-socket\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.937808 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.937718 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-data\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.938383 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.938365 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istiod-ca-cert\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.940242 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.940196 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-envoy\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.941862 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.941835 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-podinfo\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.961510 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.961480 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cmmg\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-kube-api-access-6cmmg\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:42.964254 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:42.964218 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-token\") pod \"e2e-ait-573837e3-openshift-default-7779f764c-crrqn\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:43.028671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.028633 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" event={"ID":"b36c2f94-365f-4e7e-89f4-2bcb24407d95","Type":"ContainerDied","Data":"d52a57f2339b24095268f828f5761a16e8e89e10eec7d9372e6103016a82e7b5"} Oct 30 00:53:43.029114 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.028695 4927 scope.go:122] "RemoveContainer" containerID="1364c8f077555e28479cdedb5e3a887332ad184e1be40e6c8d46e1f6dd20cedb" Oct 30 00:53:43.029114 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.028847 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h" Oct 30 00:53:43.034105 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.034066 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" event={"ID":"09475bc1-6998-4664-99e5-e56ee4aa8f3b","Type":"ContainerDied","Data":"f8a977a30bad2ad20dbe8ceb209df3db138bebb38182dae743410140e136aecf"} Oct 30 00:53:43.034740 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.034144 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t" Oct 30 00:53:43.044670 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.044257 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" event={"ID":"61512ccc-12bf-4206-9163-d11212cdba6d","Type":"ContainerStarted","Data":"c4ce26da8ee4c05661ff1eb98f54f3498b3d3452b5fce560780d5981db35019b"} Oct 30 00:53:43.044949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.044721 4927 patch_prober.go:29] interesting pod/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:53:43.044949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.044773 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:53:43.049123 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.049099 4927 scope.go:122] "RemoveContainer" containerID="3449b97284564b8a90c1ed18d86f557533a3b2125a5e35cde1776bfba0c8fc3b" Oct 30 00:53:43.083811 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.077026 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:43.131683 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.131661 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h"] Oct 30 00:53:43.145006 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.144938 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74b07396-5bcdf47756-nvh8h"] Oct 30 00:53:43.152187 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.152151 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t"] Oct 30 00:53:43.156552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.156518 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74b07396-69664fc94c-qdt6t"] Oct 30 00:53:43.820667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.817654 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:43.820667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.817728 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:43.913635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.911848 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09475bc1-6998-4664-99e5-e56ee4aa8f3b" path="/var/lib/kubelet/pods/09475bc1-6998-4664-99e5-e56ee4aa8f3b/volumes" Oct 30 00:53:43.970720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.933446 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36c2f94-365f-4e7e-89f4-2bcb24407d95" path="/var/lib/kubelet/pods/b36c2f94-365f-4e7e-89f4-2bcb24407d95/volumes" Oct 30 00:53:43.970720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.934053 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx"] Oct 30 00:53:43.970720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:43.934338 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" podUID="60fa29a9-51cb-4c79-a1d4-a6a802ede342" containerName="maas-api" containerID="cri-o://bb101d9016118cd99c7ddf552a82afcc828b3fc42f5fa95acbf478ea3bc78aed" gracePeriod=30 Oct 30 00:53:44.524613 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:44.505490 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:44.524613 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:44.505613 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:44.524613 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:44.505697 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:44.783886 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:44.780978 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn"] Oct 30 00:53:44.871600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:44.836188 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:44.872456 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:44.872403 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:44.996780 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:44.996722 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:45.020252 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:45.020206 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 00:53:45.042848 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:45.042257 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" podUID="60fa29a9-51cb-4c79-a1d4-a6a802ede342" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.110:8443/health\": dial tcp 10.132.0.110:8443: connect: connection refused" Oct 30 00:53:45.122601 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:45.113813 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:45.299846 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:45.299749 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:45.456195 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:45.455620 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 00:53:45.885614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:45.867412 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:45.885614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:45.867477 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:46.004652 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.004437 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:46.167705 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.147821 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 00:53:46.371876 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.371836 4927 generic.go:356] "Generic (PLEG): container finished" podID="60fa29a9-51cb-4c79-a1d4-a6a802ede342" containerID="bb101d9016118cd99c7ddf552a82afcc828b3fc42f5fa95acbf478ea3bc78aed" exitCode=0 Oct 30 00:53:46.434589 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.434479 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:53:46.434958 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.434891 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:53:46.435129 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.435104 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" event={"ID":"60fa29a9-51cb-4c79-a1d4-a6a802ede342","Type":"ContainerDied","Data":"bb101d9016118cd99c7ddf552a82afcc828b3fc42f5fa95acbf478ea3bc78aed"} Oct 30 00:53:46.435396 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.435377 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 00:53:46.435522 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.435505 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" event={"ID":"f0fcc8c5-19f6-4599-8038-f367ab8828a5","Type":"ContainerStarted","Data":"30e5fdb5d2752b87feb3d1693c86a30916f1792d67976487b03ddf324bd161fb"} Oct 30 00:53:46.435960 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.435652 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:53:46.505801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.453718 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:46.521600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.519316 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" podStartSLOduration=8.881979358 podStartE2EDuration="10.519290259s" totalImagesPullingTime="1.637310901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:46.505255063 +0000 UTC m=+1165.077181290" watchObservedRunningTime="2026-10-30 00:53:46.519290259 +0000 UTC m=+1165.091216482" Oct 30 00:53:46.541775 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.541743 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 00:53:46.812340 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.812300 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:46.812510 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.812363 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:46.847947 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:46.847921 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:53:47.082650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.080838 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60fa29a9-51cb-4c79-a1d4-a6a802ede342-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fa29a9-51cb-4c79-a1d4-a6a802ede342-openshift-service-ca\") pod \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " Oct 30 00:53:47.082650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.081070 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60fa29a9-51cb-4c79-a1d4-a6a802ede342-kube-api-access-h8nx4\" (UniqueName: \"kubernetes.io/projected/60fa29a9-51cb-4c79-a1d4-a6a802ede342-kube-api-access-h8nx4\") pod \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " Oct 30 00:53:47.082650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.081206 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-metrics-certs\") pod \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " Oct 30 00:53:47.082650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.081239 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-maas-api-tls\") pod \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\" (UID: \"60fa29a9-51cb-4c79-a1d4-a6a802ede342\") " Oct 30 00:53:47.083339 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.083308 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60fa29a9-51cb-4c79-a1d4-a6a802ede342-openshift-service-ca" pod "60fa29a9-51cb-4c79-a1d4-a6a802ede342" (UID: "60fa29a9-51cb-4c79-a1d4-a6a802ede342"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:47.101494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.098009 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-metrics-certs" pod "60fa29a9-51cb-4c79-a1d4-a6a802ede342" (UID: "60fa29a9-51cb-4c79-a1d4-a6a802ede342"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:47.101494 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.098471 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-maas-api-tls" pod "60fa29a9-51cb-4c79-a1d4-a6a802ede342" (UID: "60fa29a9-51cb-4c79-a1d4-a6a802ede342"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:47.108490 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.108439 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fa29a9-51cb-4c79-a1d4-a6a802ede342-kube-api-access-h8nx4" pod "60fa29a9-51cb-4c79-a1d4-a6a802ede342" (UID: "60fa29a9-51cb-4c79-a1d4-a6a802ede342"). InnerVolumeSpecName "kube-api-access-h8nx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:47.183585 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.183536 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fa29a9-51cb-4c79-a1d4-a6a802ede342-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:47.183585 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.183591 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8nx4\" (UniqueName: \"kubernetes.io/projected/60fa29a9-51cb-4c79-a1d4-a6a802ede342-kube-api-access-h8nx4\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:47.183802 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.183608 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:47.183802 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.183620 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fa29a9-51cb-4c79-a1d4-a6a802ede342-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:47.439769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.439693 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" event={"ID":"61512ccc-12bf-4206-9163-d11212cdba6d","Type":"ContainerStarted","Data":"477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b"} Oct 30 00:53:47.442881 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.442848 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" event={"ID":"f0fcc8c5-19f6-4599-8038-f367ab8828a5","Type":"ContainerStarted","Data":"3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3"} Oct 30 00:53:47.445214 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.445183 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" Oct 30 00:53:47.445214 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.445214 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx" event={"ID":"60fa29a9-51cb-4c79-a1d4-a6a802ede342","Type":"ContainerDied","Data":"7d7fe3b4953aad8ba7f8878d6d33d53965c4dbf402396feb87e290aaf90b6e54"} Oct 30 00:53:47.445536 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.445385 4927 scope.go:122] "RemoveContainer" containerID="bb101d9016118cd99c7ddf552a82afcc828b3fc42f5fa95acbf478ea3bc78aed" Oct 30 00:53:47.502547 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.502492 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" podStartSLOduration=6.502475443 podStartE2EDuration="6.502475443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:47.498145924 +0000 UTC m=+1166.070072186" watchObservedRunningTime="2026-10-30 00:53:47.502475443 +0000 UTC m=+1166.074401740" Oct 30 00:53:47.536566 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.536526 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx"] Oct 30 00:53:47.541799 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.541759 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4e6c558-8b9bdd97c-5flmx"] Oct 30 00:53:47.573658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.573622 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60fa29a9-51cb-4c79-a1d4-a6a802ede342" path="/var/lib/kubelet/pods/60fa29a9-51cb-4c79-a1d4-a6a802ede342/volumes" Oct 30 00:53:47.822289 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.822248 4927 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:53:47.822693 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:47.822559 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 00:53:48.026247 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.026217 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:53:48.078146 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.078109 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:48.086808 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.086771 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:48.127391 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.127341 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-podinfo\") pod \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " Oct 30 00:53:48.127654 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.127474 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istiod-ca-cert\") pod \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " Oct 30 00:53:48.127654 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.127558 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-certs\") pod \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " Oct 30 00:53:48.127793 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.127695 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-kube-api-access-j7wb7\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-kube-api-access-j7wb7\") pod \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " Oct 30 00:53:48.127793 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.127730 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-token\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-token\") pod \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " Oct 30 00:53:48.127900 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.127830 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-socket\") pod \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " Oct 30 00:53:48.128093 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.128051 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-certs" pod "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" (UID: "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:48.128268 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.128248 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-socket" pod "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" (UID: "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:48.128865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.128835 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.128989 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.128869 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.128989 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.128942 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istiod-ca-cert" pod "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" (UID: "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:48.130564 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.130508 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-podinfo" pod "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" (UID: "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:53:48.132528 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.132449 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-token" pod "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" (UID: "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:48.133237 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.133206 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-kube-api-access-j7wb7" pod "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" (UID: "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721"). InnerVolumeSpecName "kube-api-access-j7wb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:48.230323 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.230285 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-envoy\") pod \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " Oct 30 00:53:48.230517 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.230338 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-data\") pod \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " Oct 30 00:53:48.230517 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.230383 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-credential-socket\") pod \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\" (UID: \"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721\") " Oct 30 00:53:48.230678 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.230655 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-data" pod "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" (UID: "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:48.230766 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.230743 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-credential-socket" pod "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" (UID: "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:48.231136 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.231114 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.231136 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.231135 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.231277 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.231146 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7wb7\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-kube-api-access-j7wb7\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.231277 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.231157 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.231277 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.231167 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.231277 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.231178 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.232614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.232557 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-envoy" pod "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" (UID: "6b3fcea8-9fbb-4cc3-909a-1ca4deb37721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:53:48.332393 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.332307 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:53:48.462063 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.462027 4927 generic.go:356] "Generic (PLEG): container finished" podID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerID="4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166" exitCode=0 Oct 30 00:53:48.462236 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.462086 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" Oct 30 00:53:48.462236 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.462103 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" event={"ID":"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721","Type":"ContainerDied","Data":"4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166"} Oct 30 00:53:48.462236 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.462164 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2" event={"ID":"6b3fcea8-9fbb-4cc3-909a-1ca4deb37721","Type":"ContainerDied","Data":"f0f2bec14bcfd20b14ee2c5e162f4cd1f7494d08aa007e8027fb53968415bb17"} Oct 30 00:53:48.462236 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.462188 4927 scope.go:122] "RemoveContainer" containerID="4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166" Oct 30 00:53:48.463355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.463197 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:48.464340 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.464311 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:53:48.476423 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.476406 4927 scope.go:122] "RemoveContainer" containerID="2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e" Oct 30 00:53:48.491090 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.491063 4927 scope.go:122] "RemoveContainer" containerID="4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166" Oct 30 00:53:48.491510 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:48.491480 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166\": container with ID starting with 4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166 not found: ID does not exist" containerID="4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166" Oct 30 00:53:48.491655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.491521 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166"} err="failed to get container status \"4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166\": rpc error: code = NotFound desc = could not find container \"4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166\": container with ID starting with 4379040a39dd162d99df131412847de451d59c89535050af1b3ad34676e68166 not found: ID does not exist" Oct 30 00:53:48.491655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.491545 4927 scope.go:122] "RemoveContainer" containerID="2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e" Oct 30 00:53:48.491937 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:53:48.491910 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e\": container with ID starting with 2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e not found: ID does not exist" containerID="2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e" Oct 30 00:53:48.492011 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.491947 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e"} err="failed to get container status \"2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e\": rpc error: code = NotFound desc = could not find container \"2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e\": container with ID starting with 2228be00998b5b3303dbdcfd36fa17fcbf45daac3b75b75f7133baadd7dd9a2e not found: ID does not exist" Oct 30 00:53:48.516252 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.516214 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2"] Oct 30 00:53:48.533722 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:48.533678 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1e708b2f-gw-openshift-default-545cdc848f-lcmj2"] Oct 30 00:53:49.048926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:49.048892 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:49.049444 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:49.049403 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:49.050312 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:49.050289 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:53:49.574880 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:49.574839 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" path="/var/lib/kubelet/pods/6b3fcea8-9fbb-4cc3-909a-1ca4deb37721/volumes" Oct 30 00:53:51.106688 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.106651 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2"] Oct 30 00:53:51.107749 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107723 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b36c2f94-365f-4e7e-89f4-2bcb24407d95" containerName="payload-processing" Oct 30 00:53:51.107749 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107746 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36c2f94-365f-4e7e-89f4-2bcb24407d95" containerName="payload-processing" Oct 30 00:53:51.107949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107768 4927 cpu_manager.go:414] "containerMap: removing container" podUID="09475bc1-6998-4664-99e5-e56ee4aa8f3b" containerName="payload-pre-processing" Oct 30 00:53:51.107949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107776 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09475bc1-6998-4664-99e5-e56ee4aa8f3b" containerName="payload-pre-processing" Oct 30 00:53:51.107949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107811 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" Oct 30 00:53:51.107949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107820 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" Oct 30 00:53:51.107949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107833 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" Oct 30 00:53:51.107949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107839 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" Oct 30 00:53:51.107949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107855 4927 cpu_manager.go:414] "containerMap: removing container" podUID="60fa29a9-51cb-4c79-a1d4-a6a802ede342" containerName="maas-api" Oct 30 00:53:51.107949 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.107862 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60fa29a9-51cb-4c79-a1d4-a6a802ede342" containerName="maas-api" Oct 30 00:53:51.108307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.108063 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="60fa29a9-51cb-4c79-a1d4-a6a802ede342" containerName="maas-api" Oct 30 00:53:51.108307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.108091 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36c2f94-365f-4e7e-89f4-2bcb24407d95" containerName="payload-processing" Oct 30 00:53:51.108307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.108103 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="09475bc1-6998-4664-99e5-e56ee4aa8f3b" containerName="payload-pre-processing" Oct 30 00:53:51.108307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.108117 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" Oct 30 00:53:51.108307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.108124 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3fcea8-9fbb-4cc3-909a-1ca4deb37721" containerName="istio-proxy" Oct 30 00:53:51.113724 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.113700 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:53:51.131801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.131769 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2"] Oct 30 00:53:51.136615 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.136558 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-573837e3\"" Oct 30 00:53:51.136818 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.136670 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-573837e3-dockercfg-qv8f8\"" Oct 30 00:53:51.175984 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.175924 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r72bj\" (UniqueName: \"kubernetes.io/projected/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-kube-api-access-r72bj\") pod \"payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2\" (UID: \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:53:51.176394 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.176363 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2\" (UID: \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:53:51.560622 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.560551 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r72bj\" (UniqueName: \"kubernetes.io/projected/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-kube-api-access-r72bj\") pod \"payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2\" (UID: \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:53:51.560931 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.560910 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2\" (UID: \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:53:51.562185 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.562155 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2\" (UID: \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:53:51.648120 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.646930 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r72bj\" (UniqueName: \"kubernetes.io/projected/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-kube-api-access-r72bj\") pod \"payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2\" (UID: \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:53:51.729828 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:51.729790 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:53:52.061216 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:52.061145 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2"] Oct 30 00:53:52.062312 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:53:52.062274 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b9eac2_3fc4_4e33_980f_d28cb14dbaa4.slice/crio-3cfa6d79e4003eabd0d7b58fda66e98e7e4e4b7b2f7a40ec943741711042a8fd WatchSource:0}: Error finding container 3cfa6d79e4003eabd0d7b58fda66e98e7e4e4b7b2f7a40ec943741711042a8fd: Status 404 returned error can't find the container with id 3cfa6d79e4003eabd0d7b58fda66e98e7e4e4b7b2f7a40ec943741711042a8fd Oct 30 00:53:52.508658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:52.508606 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" event={"ID":"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4","Type":"ContainerStarted","Data":"c1dcef79f4bd2c1d25fd28720bcf06d1e8c81c9860be86ddf379410e5e349c53"} Oct 30 00:53:52.509131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:52.508711 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" event={"ID":"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4","Type":"ContainerStarted","Data":"3cfa6d79e4003eabd0d7b58fda66e98e7e4e4b7b2f7a40ec943741711042a8fd"} Oct 30 00:53:52.509131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:52.508738 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:53:52.561773 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:52.561710 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" podStartSLOduration=1.561690059 podStartE2EDuration="1.561690059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:52.558072909 +0000 UTC m=+1171.129999132" watchObservedRunningTime="2026-10-30 00:53:52.561690059 +0000 UTC m=+1171.133616282" Oct 30 00:53:53.467443 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:53:53.467413 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 00:54:03.689324 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:03.689286 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:54:04.445503 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:54:04.445466 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:54:05.014375 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.014324 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z"] Oct 30 00:54:05.048603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.045992 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.067594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.059880 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z"] Oct 30 00:54:05.067594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.062611 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-573837e3-serving-cert\"" Oct 30 00:54:05.158320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.158270 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-openshift-service-ca\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.158532 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.158411 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-metrics-certs\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.158532 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.158478 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-maas-api-tls\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.158731 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.158705 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h95s\" (UniqueName: \"kubernetes.io/projected/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-kube-api-access-4h95s\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.260176 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.260141 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h95s\" (UniqueName: \"kubernetes.io/projected/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-kube-api-access-4h95s\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.260390 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.260377 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-openshift-service-ca\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.260441 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.260420 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-metrics-certs\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.260473 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.260464 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-maas-api-tls\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.261193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.261167 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-openshift-service-ca\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.263201 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.263173 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-metrics-certs\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.263289 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.263173 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-maas-api-tls\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.282416 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.282338 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h95s\" (UniqueName: \"kubernetes.io/projected/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-kube-api-access-4h95s\") pod \"maas-api-e2e-ait-573837e3-dfdd978b9-m485z\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.428650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.422280 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:05.826618 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:05.826532 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z"] Oct 30 00:54:05.827166 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:54:05.827133 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfd237c_7ff9_4ce0_9984_b06818ca8b37.slice/crio-4ebec1ddee20d8879c16c11909cd3a605e67cb387004a2c963bb2a7225cb7463 WatchSource:0}: Error finding container 4ebec1ddee20d8879c16c11909cd3a605e67cb387004a2c963bb2a7225cb7463: Status 404 returned error can't find the container with id 4ebec1ddee20d8879c16c11909cd3a605e67cb387004a2c963bb2a7225cb7463 Oct 30 00:54:06.720307 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:06.720216 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" event={"ID":"bbfd237c-7ff9-4ce0-9984-b06818ca8b37","Type":"ContainerStarted","Data":"4ebec1ddee20d8879c16c11909cd3a605e67cb387004a2c963bb2a7225cb7463"} Oct 30 00:54:07.730378 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:07.730343 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" event={"ID":"bbfd237c-7ff9-4ce0-9984-b06818ca8b37","Type":"ContainerStarted","Data":"abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef"} Oct 30 00:54:07.730864 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:07.730479 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:12.396594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.387956 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" podStartSLOduration=7.845567547 podStartE2EDuration="8.387930699s" totalImagesPullingTime="542.363152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:07.777396253 +0000 UTC m=+1186.349322476" watchObservedRunningTime="2026-10-30 00:54:12.387930699 +0000 UTC m=+1190.959856922" Oct 30 00:54:12.431936 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.431875 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8"] Oct 30 00:54:12.616129 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.590564 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:12.616129 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.600815 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8"] Oct 30 00:54:12.634868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.632449 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-60e8f4-dockercfg-2px8s\"" Oct 30 00:54:12.634868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.632535 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-60e8f4\"" Oct 30 00:54:12.762392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.762333 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97acea83-49aa-4c90-bf1c-312dc1dd7d02-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8\" (UID: \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:12.762645 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.762621 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjvg\" (UniqueName: \"kubernetes.io/projected/97acea83-49aa-4c90-bf1c-312dc1dd7d02-kube-api-access-xkjvg\") pod \"payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8\" (UID: \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:12.863958 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.863905 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjvg\" (UniqueName: \"kubernetes.io/projected/97acea83-49aa-4c90-bf1c-312dc1dd7d02-kube-api-access-xkjvg\") pod \"payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8\" (UID: \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:12.864265 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.864248 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97acea83-49aa-4c90-bf1c-312dc1dd7d02-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8\" (UID: \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:12.864993 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.864974 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97acea83-49aa-4c90-bf1c-312dc1dd7d02-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8\" (UID: \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:12.889817 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.889785 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjvg\" (UniqueName: \"kubernetes.io/projected/97acea83-49aa-4c90-bf1c-312dc1dd7d02-kube-api-access-xkjvg\") pod \"payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8\" (UID: \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:12.995326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:12.995127 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:13.421261 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:13.421226 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8"] Oct 30 00:54:13.422249 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:54:13.422224 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97acea83_49aa_4c90_bf1c_312dc1dd7d02.slice/crio-59dbf460e1e42e7281b999d778b9007e78a5c43135145b45dec984467746eda0 WatchSource:0}: Error finding container 59dbf460e1e42e7281b999d778b9007e78a5c43135145b45dec984467746eda0: Status 404 returned error can't find the container with id 59dbf460e1e42e7281b999d778b9007e78a5c43135145b45dec984467746eda0 Oct 30 00:54:13.758115 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:13.758033 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:13.796276 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:13.796236 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" event={"ID":"97acea83-49aa-4c90-bf1c-312dc1dd7d02","Type":"ContainerStarted","Data":"06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc"} Oct 30 00:54:13.796458 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:13.796300 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" event={"ID":"97acea83-49aa-4c90-bf1c-312dc1dd7d02","Type":"ContainerStarted","Data":"59dbf460e1e42e7281b999d778b9007e78a5c43135145b45dec984467746eda0"} Oct 30 00:54:13.796458 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:13.796322 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:21.838457 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:21.838427 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:54:21.839111 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:21.838482 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:54:22.087869 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.087810 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" podStartSLOduration=10.087790757 podStartE2EDuration="10.087790757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:13.840479636 +0000 UTC m=+1192.412405870" watchObservedRunningTime="2026-10-30 00:54:22.087790757 +0000 UTC m=+1200.659716978" Oct 30 00:54:22.088648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.088466 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5"] Oct 30 00:54:22.094973 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.094947 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.114130 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.114085 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-envoy\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.114328 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.114209 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-socket\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.114328 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.114240 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-podinfo\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.114328 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.114319 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43321879-3683-44ec-b6b6-11820f4ad9f2-istiod-ca-cert\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.114552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.114421 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8pd\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-kube-api-access-pp8pd\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.114552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.114445 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-token\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.114552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.114486 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-credential-socket\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.114552 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.114532 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-certs\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.114935 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.114562 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-data\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.120388 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.120354 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5"] Oct 30 00:54:22.123261 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.123238 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d8090513-openshift-default-dockercfg-8vxq7\"" Oct 30 00:54:22.217830 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.217787 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8pd\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-kube-api-access-pp8pd\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218046 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.217834 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-token\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218046 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.217879 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-credential-socket\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218046 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.217925 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-certs\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218046 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.217953 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-data\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218256 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.218194 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-envoy\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218317 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.218304 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-socket\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218376 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.218335 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-podinfo\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218376 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.218358 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-certs\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.218377 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-credential-socket\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.218437 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-data\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218645 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.218525 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43321879-3683-44ec-b6b6-11820f4ad9f2-istiod-ca-cert\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.218712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.218699 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-socket\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.219104 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.219084 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43321879-3683-44ec-b6b6-11820f4ad9f2-istiod-ca-cert\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.220524 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.220508 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-envoy\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.220984 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.220962 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-podinfo\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.240299 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.240268 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-token\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.240443 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.240357 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8pd\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-kube-api-access-pp8pd\") pod \"e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:22.415935 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:22.415893 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:23.017138 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:23.017100 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5"] Oct 30 00:54:23.017138 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:54:23.017123 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43321879_3683_44ec_b6b6_11820f4ad9f2.slice/crio-e0a7b5999710d181132816824e9e89d5f77fd1143817b01afc07d8c0cbd80a1b WatchSource:0}: Error finding container e0a7b5999710d181132816824e9e89d5f77fd1143817b01afc07d8c0cbd80a1b: Status 404 returned error can't find the container with id e0a7b5999710d181132816824e9e89d5f77fd1143817b01afc07d8c0cbd80a1b Oct 30 00:54:23.020424 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:23.020384 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:23.020527 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:23.020455 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:23.020527 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:23.020485 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:23.891283 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:23.891239 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" event={"ID":"43321879-3683-44ec-b6b6-11820f4ad9f2","Type":"ContainerStarted","Data":"11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f"} Oct 30 00:54:23.891541 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:23.891309 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" event={"ID":"43321879-3683-44ec-b6b6-11820f4ad9f2","Type":"ContainerStarted","Data":"e0a7b5999710d181132816824e9e89d5f77fd1143817b01afc07d8c0cbd80a1b"} Oct 30 00:54:23.970831 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:23.970760 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" podStartSLOduration=1.970743301 podStartE2EDuration="1.970743301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:23.954797148 +0000 UTC m=+1202.526723370" watchObservedRunningTime="2026-10-30 00:54:23.970743301 +0000 UTC m=+1202.542669523" Oct 30 00:54:24.417285 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:24.417247 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:24.423009 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:24.422982 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:24.809702 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:24.809614 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 00:54:24.904492 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:24.904458 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:24.905767 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:24.905723 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:54:29.961138 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:29.961099 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm"] Oct 30 00:54:29.967478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:29.967451 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.012525 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.012487 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-60e8f4-serving-cert\"" Oct 30 00:54:30.026728 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.026696 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm"] Oct 30 00:54:30.070082 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.070046 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a52d418-6bef-48df-841e-5b55dd4e42ac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.070269 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.070092 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.070269 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.070217 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.070372 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.070350 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqfw4\" (UniqueName: \"kubernetes.io/projected/5a52d418-6bef-48df-841e-5b55dd4e42ac-kube-api-access-wqfw4\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.172114 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.172071 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a52d418-6bef-48df-841e-5b55dd4e42ac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.172309 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.172124 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.172309 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.172176 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.172309 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.172251 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqfw4\" (UniqueName: \"kubernetes.io/projected/5a52d418-6bef-48df-841e-5b55dd4e42ac-kube-api-access-wqfw4\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.172851 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.172826 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a52d418-6bef-48df-841e-5b55dd4e42ac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.198241 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.198203 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.198479 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.198247 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.257655 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.248198 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqfw4\" (UniqueName: \"kubernetes.io/projected/5a52d418-6bef-48df-841e-5b55dd4e42ac-kube-api-access-wqfw4\") pod \"maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.334635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.331930 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:30.817651 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:30.814856 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm"] Oct 30 00:54:31.477187 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:31.468601 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" event={"ID":"5a52d418-6bef-48df-841e-5b55dd4e42ac","Type":"ContainerStarted","Data":"d4b70fd7a9e02d4b03fa240d277bd405f1bd297294d78e88bccb627b115ea94b"} Oct 30 00:54:32.479144 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:32.479049 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" event={"ID":"5a52d418-6bef-48df-841e-5b55dd4e42ac","Type":"ContainerStarted","Data":"0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037"} Oct 30 00:54:32.479536 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:32.479162 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:32.522661 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:32.522527 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" podStartSLOduration=2.9867953529999998 podStartE2EDuration="3.522510293s" totalImagesPullingTime="535.71494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:32.519768078 +0000 UTC m=+1211.091694303" watchObservedRunningTime="2026-10-30 00:54:32.522510293 +0000 UTC m=+1211.094436561" Oct 30 00:54:34.447930 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:54:34.447886 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:54:34.503740 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:34.503699 4927 patch_prober.go:29] interesting pod/e2e-ait-573837e3-openshift-default-7779f764c-crrqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:54:34.503930 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:34.503763 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:54:37.528500 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:37.528466 4927 generic.go:356] "Generic (PLEG): container finished" podID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerID="3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3" exitCode=0 Oct 30 00:54:37.529120 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:37.528554 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" event={"ID":"f0fcc8c5-19f6-4599-8038-f367ab8828a5","Type":"ContainerDied","Data":"3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3"} Oct 30 00:54:37.529261 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:37.529242 4927 scope.go:122] "RemoveContainer" containerID="3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3" Oct 30 00:54:37.531132 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:37.531099 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:37.531238 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:37.531174 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:37.531238 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:37.531211 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:38.508189 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:38.508045 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 00:54:38.540173 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:38.540140 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" event={"ID":"f0fcc8c5-19f6-4599-8038-f367ab8828a5","Type":"ContainerStarted","Data":"bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54"} Oct 30 00:54:39.082596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:39.078747 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:54:39.082596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:39.078896 4927 patch_prober.go:29] interesting pod/e2e-ait-573837e3-openshift-default-7779f764c-crrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:54:39.082596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:39.078944 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:54:40.093599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:40.078035 4927 patch_prober.go:29] interesting pod/e2e-ait-573837e3-openshift-default-7779f764c-crrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:54:40.093599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:40.078119 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:54:41.080094 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:41.080048 4927 patch_prober.go:29] interesting pod/e2e-ait-573837e3-openshift-default-7779f764c-crrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:54:41.080392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:41.080354 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:54:41.460872 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:41.460826 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2"] Oct 30 00:54:42.152603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:42.140832 4927 patch_prober.go:29] interesting pod/e2e-ait-573837e3-openshift-default-7779f764c-crrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:54:42.152603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:42.140900 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:54:43.078596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:43.078047 4927 patch_prober.go:29] interesting pod/e2e-ait-573837e3-openshift-default-7779f764c-crrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:54:43.078596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:43.078116 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:54:44.094656 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:44.094614 4927 patch_prober.go:29] interesting pod/e2e-ait-573837e3-openshift-default-7779f764c-crrqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:54:44.095753 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:44.095272 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 00:54:44.589992 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:44.583887 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z"] Oct 30 00:54:44.589992 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:44.584154 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" podUID="bbfd237c-7ff9-4ce0-9984-b06818ca8b37" containerName="maas-api" containerID="cri-o://abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef" gracePeriod=30 Oct 30 00:54:45.557688 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.550523 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:45.676832 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.676799 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-kube-api-access-4h95s\" (UniqueName: \"kubernetes.io/projected/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-kube-api-access-4h95s\") pod \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " Oct 30 00:54:45.677043 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.676862 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-metrics-certs\") pod \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " Oct 30 00:54:45.677043 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.676978 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-openshift-service-ca\") pod \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " Oct 30 00:54:45.677043 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.677010 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-maas-api-tls\") pod \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\" (UID: \"bbfd237c-7ff9-4ce0-9984-b06818ca8b37\") " Oct 30 00:54:45.677485 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.677423 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-openshift-service-ca" pod "bbfd237c-7ff9-4ce0-9984-b06818ca8b37" (UID: "bbfd237c-7ff9-4ce0-9984-b06818ca8b37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:45.678956 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.678926 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:45.679726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.679697 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-kube-api-access-4h95s" pod "bbfd237c-7ff9-4ce0-9984-b06818ca8b37" (UID: "bbfd237c-7ff9-4ce0-9984-b06818ca8b37"). InnerVolumeSpecName "kube-api-access-4h95s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:45.679838 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.679777 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-metrics-certs" pod "bbfd237c-7ff9-4ce0-9984-b06818ca8b37" (UID: "bbfd237c-7ff9-4ce0-9984-b06818ca8b37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:54:45.679879 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.679858 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-maas-api-tls" pod "bbfd237c-7ff9-4ce0-9984-b06818ca8b37" (UID: "bbfd237c-7ff9-4ce0-9984-b06818ca8b37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:54:45.706226 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.706192 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:54:45.706745 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.706716 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:54:45.709267 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.709237 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:54:45.714117 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.714085 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk"] Oct 30 00:54:45.715302 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.715279 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bbfd237c-7ff9-4ce0-9984-b06818ca8b37" containerName="maas-api" Oct 30 00:54:45.715302 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.715302 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbfd237c-7ff9-4ce0-9984-b06818ca8b37" containerName="maas-api" Oct 30 00:54:45.715540 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.715524 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbfd237c-7ff9-4ce0-9984-b06818ca8b37" containerName="maas-api" Oct 30 00:54:45.719944 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.719921 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:45.738825 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.736334 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d8090513-serving-cert\"" Oct 30 00:54:45.776112 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.757102 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk"] Oct 30 00:54:45.776112 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.770023 4927 generic.go:356] "Generic (PLEG): container finished" podID="bbfd237c-7ff9-4ce0-9984-b06818ca8b37" containerID="abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef" exitCode=0 Oct 30 00:54:45.776112 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.774719 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" Oct 30 00:54:45.776112 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.774904 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" event={"ID":"bbfd237c-7ff9-4ce0-9984-b06818ca8b37","Type":"ContainerDied","Data":"abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef"} Oct 30 00:54:45.776112 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.774954 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z" event={"ID":"bbfd237c-7ff9-4ce0-9984-b06818ca8b37","Type":"ContainerDied","Data":"4ebec1ddee20d8879c16c11909cd3a605e67cb387004a2c963bb2a7225cb7463"} Oct 30 00:54:45.776112 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.774985 4927 scope.go:122] "RemoveContainer" containerID="abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef" Oct 30 00:54:45.808260 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.808143 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h95s\" (UniqueName: \"kubernetes.io/projected/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-kube-api-access-4h95s\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:45.808260 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.808177 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:45.808260 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.808196 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfd237c-7ff9-4ce0-9984-b06818ca8b37-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:45.915284 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.915247 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2909829-1690-436a-819e-1c21db428990-openshift-service-ca\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:45.915444 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.915315 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lggf\" (UniqueName: \"kubernetes.io/projected/f2909829-1690-436a-819e-1c21db428990-kube-api-access-9lggf\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:45.915493 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.915454 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-metrics-certs\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:45.915628 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.915610 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-maas-api-tls\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:45.923305 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.923272 4927 scope.go:122] "RemoveContainer" containerID="abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef" Oct 30 00:54:45.923889 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:54:45.923781 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef\": container with ID starting with abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef not found: ID does not exist" containerID="abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef" Oct 30 00:54:45.923889 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.923825 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef"} err="failed to get container status \"abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef\": rpc error: code = NotFound desc = could not find container \"abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef\": container with ID starting with abe16c94570e45207dbd5a26149ac21f6f5e23cae5341a85de0fe052941fbeef not found: ID does not exist" Oct 30 00:54:45.997241 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:45.997127 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z"] Oct 30 00:54:46.002223 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.002192 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-573837e3-dfdd978b9-m485z"] Oct 30 00:54:46.017085 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.017053 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2909829-1690-436a-819e-1c21db428990-openshift-service-ca\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:46.017262 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.017095 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lggf\" (UniqueName: \"kubernetes.io/projected/f2909829-1690-436a-819e-1c21db428990-kube-api-access-9lggf\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:46.017262 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.017149 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-metrics-certs\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:46.017262 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.017219 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-maas-api-tls\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:46.017762 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.017734 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2909829-1690-436a-819e-1c21db428990-openshift-service-ca\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:46.019804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.019773 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-metrics-certs\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:46.019935 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.019853 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-maas-api-tls\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:46.039608 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.039561 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lggf\" (UniqueName: \"kubernetes.io/projected/f2909829-1690-436a-819e-1c21db428990-kube-api-access-9lggf\") pod \"maas-api-e2e-mt-d8090513-784b489945-jzgxk\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:46.280716 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.280621 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:46.462121 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.462072 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" podUID="81b9eac2-3fc4-4e33-980f-d28cb14dbaa4" containerName="payload-pre-processing" containerID="cri-o://c1dcef79f4bd2c1d25fd28720bcf06d1e8c81c9860be86ddf379410e5e349c53" gracePeriod=25 Oct 30 00:54:46.595595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.595548 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk"] Oct 30 00:54:46.842119 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.841983 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" event={"ID":"f2909829-1690-436a-819e-1c21db428990","Type":"ContainerStarted","Data":"099195db54a23b5f39c17ea3ef5c524f8551487eb15cf0057ddf2b2b3e4cb27a"} Oct 30 00:54:46.844231 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.844201 4927 generic.go:356] "Generic (PLEG): container finished" podID="81b9eac2-3fc4-4e33-980f-d28cb14dbaa4" containerID="c1dcef79f4bd2c1d25fd28720bcf06d1e8c81c9860be86ddf379410e5e349c53" exitCode=0 Oct 30 00:54:46.844393 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.844271 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" event={"ID":"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4","Type":"ContainerDied","Data":"c1dcef79f4bd2c1d25fd28720bcf06d1e8c81c9860be86ddf379410e5e349c53"} Oct 30 00:54:46.859896 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.859868 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:54:46.962350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.962307 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-plugins-config-volume\") pod \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\" (UID: \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\") " Oct 30 00:54:46.962531 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.962374 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-kube-api-access-r72bj\" (UniqueName: \"kubernetes.io/projected/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-kube-api-access-r72bj\") pod \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\" (UID: \"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4\") " Oct 30 00:54:46.962931 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.962901 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-plugins-config-volume" pod "81b9eac2-3fc4-4e33-980f-d28cb14dbaa4" (UID: "81b9eac2-3fc4-4e33-980f-d28cb14dbaa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:46.964154 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.964118 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:46.975208 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:46.975166 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-kube-api-access-r72bj" pod "81b9eac2-3fc4-4e33-980f-d28cb14dbaa4" (UID: "81b9eac2-3fc4-4e33-980f-d28cb14dbaa4"). InnerVolumeSpecName "kube-api-access-r72bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:47.066102 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:47.066067 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r72bj\" (UniqueName: \"kubernetes.io/projected/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4-kube-api-access-r72bj\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:47.652885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:47.652835 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbfd237c-7ff9-4ce0-9984-b06818ca8b37" path="/var/lib/kubelet/pods/bbfd237c-7ff9-4ce0-9984-b06818ca8b37/volumes" Oct 30 00:54:47.654421 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:47.654395 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn"] Oct 30 00:54:48.337121 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:48.337068 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" event={"ID":"f2909829-1690-436a-819e-1c21db428990","Type":"ContainerStarted","Data":"84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb"} Oct 30 00:54:48.389858 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:48.389826 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:48.482636 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:48.477974 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" Oct 30 00:54:48.482636 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:48.479392 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2" event={"ID":"81b9eac2-3fc4-4e33-980f-d28cb14dbaa4","Type":"ContainerDied","Data":"3cfa6d79e4003eabd0d7b58fda66e98e7e4e4b7b2f7a40ec943741711042a8fd"} Oct 30 00:54:48.482636 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:48.479483 4927 scope.go:122] "RemoveContainer" containerID="c1dcef79f4bd2c1d25fd28720bcf06d1e8c81c9860be86ddf379410e5e349c53" Oct 30 00:54:48.534124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:48.534076 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" containerID="cri-o://bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54" gracePeriod=30 Oct 30 00:54:48.822856 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:48.822708 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" podStartSLOduration=3.470619317 podStartE2EDuration="3.822686138s" totalImagesPullingTime="352.066821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:48.540300917 +0000 UTC m=+1227.112227140" watchObservedRunningTime="2026-10-30 00:54:48.822686138 +0000 UTC m=+1227.394612361" Oct 30 00:54:48.849034 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:48.848626 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2"] Oct 30 00:54:48.924786 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:48.894690 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-573837e3-6d645bbdff-dpfv2"] Oct 30 00:54:49.598122 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:49.598081 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b9eac2-3fc4-4e33-980f-d28cb14dbaa4" path="/var/lib/kubelet/pods/81b9eac2-3fc4-4e33-980f-d28cb14dbaa4/volumes" Oct 30 00:54:53.799432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.799404 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:54:53.839511 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.839403 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk"] Oct 30 00:54:53.840728 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.840701 4927 cpu_manager.go:414] "containerMap: removing container" podUID="81b9eac2-3fc4-4e33-980f-d28cb14dbaa4" containerName="payload-pre-processing" Oct 30 00:54:53.840728 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.840723 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b9eac2-3fc4-4e33-980f-d28cb14dbaa4" containerName="payload-pre-processing" Oct 30 00:54:53.840937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.840745 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" Oct 30 00:54:53.840937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.840751 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" Oct 30 00:54:53.840937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.840765 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" Oct 30 00:54:53.840937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.840770 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" Oct 30 00:54:53.840937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.840896 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b9eac2-3fc4-4e33-980f-d28cb14dbaa4" containerName="payload-pre-processing" Oct 30 00:54:53.840937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.840907 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" Oct 30 00:54:53.840937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.840915 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerName="istio-proxy" Oct 30 00:54:53.857951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.857895 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:54:53.880627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.877098 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-bea3ef\"" Oct 30 00:54:53.880627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.877787 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-bea3ef-dockercfg-qnn8f\"" Oct 30 00:54:53.880627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.879015 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk"] Oct 30 00:54:53.915001 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.914960 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl"] Oct 30 00:54:53.915243 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.915142 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-token\") pod \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " Oct 30 00:54:53.915315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.915244 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-podinfo\") pod \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " Oct 30 00:54:53.915315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.915298 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istiod-ca-cert\") pod \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " Oct 30 00:54:53.915526 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.915506 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-certs\") pod \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " Oct 30 00:54:53.915624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.915544 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-kube-api-access-6cmmg\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-kube-api-access-6cmmg\") pod \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " Oct 30 00:54:53.915624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.915599 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-socket\") pod \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " Oct 30 00:54:53.915750 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.915650 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-data\") pod \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " Oct 30 00:54:53.915806 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.915793 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-credential-socket\") pod \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " Oct 30 00:54:53.915874 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.915852 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-envoy\") pod \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\" (UID: \"f0fcc8c5-19f6-4599-8038-f367ab8828a5\") " Oct 30 00:54:53.916260 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.916229 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk\" (UID: \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:54:53.916807 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.916778 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92ssw\" (UniqueName: \"kubernetes.io/projected/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-kube-api-access-92ssw\") pod \"payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk\" (UID: \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:54:53.918134 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.918089 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-credential-socket" pod "f0fcc8c5-19f6-4599-8038-f367ab8828a5" (UID: "f0fcc8c5-19f6-4599-8038-f367ab8828a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:53.918257 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.918207 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-token" pod "f0fcc8c5-19f6-4599-8038-f367ab8828a5" (UID: "f0fcc8c5-19f6-4599-8038-f367ab8828a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:53.918677 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.918470 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-socket" pod "f0fcc8c5-19f6-4599-8038-f367ab8828a5" (UID: "f0fcc8c5-19f6-4599-8038-f367ab8828a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:53.918885 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.918854 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-certs" pod "f0fcc8c5-19f6-4599-8038-f367ab8828a5" (UID: "f0fcc8c5-19f6-4599-8038-f367ab8828a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:53.918975 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.918892 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-data" pod "f0fcc8c5-19f6-4599-8038-f367ab8828a5" (UID: "f0fcc8c5-19f6-4599-8038-f367ab8828a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:53.919143 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.919099 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istiod-ca-cert" pod "f0fcc8c5-19f6-4599-8038-f367ab8828a5" (UID: "f0fcc8c5-19f6-4599-8038-f367ab8828a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:53.922070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.922041 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-podinfo" pod "f0fcc8c5-19f6-4599-8038-f367ab8828a5" (UID: "f0fcc8c5-19f6-4599-8038-f367ab8828a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:54:53.922178 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.922116 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-kube-api-access-6cmmg" pod "f0fcc8c5-19f6-4599-8038-f367ab8828a5" (UID: "f0fcc8c5-19f6-4599-8038-f367ab8828a5"). InnerVolumeSpecName "kube-api-access-6cmmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:53.922327 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.922205 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-envoy" pod "f0fcc8c5-19f6-4599-8038-f367ab8828a5" (UID: "f0fcc8c5-19f6-4599-8038-f367ab8828a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:53.922462 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:53.922448 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.103950 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92ssw\" (UniqueName: \"kubernetes.io/projected/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-kube-api-access-92ssw\") pod \"payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk\" (UID: \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104104 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk\" (UID: \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104590 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104613 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104627 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104639 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104655 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cmmg\" (UniqueName: \"kubernetes.io/projected/f0fcc8c5-19f6-4599-8038-f367ab8828a5-kube-api-access-6cmmg\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104670 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104684 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104698 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.104710 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcc8c5-19f6-4599-8038-f367ab8828a5-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:54:54.162610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.105628 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk\" (UID: \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:54:54.193674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.192203 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92ssw\" (UniqueName: \"kubernetes.io/projected/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-kube-api-access-92ssw\") pod \"payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk\" (UID: \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:54:54.193674 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.192541 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:54:54.232642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.211326 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz49g\" (UniqueName: \"kubernetes.io/projected/816fb468-458b-4597-8da0-5fe17e7dc3d8-kube-api-access-fz49g\") pod \"payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl\" (UID: \"816fb468-458b-4597-8da0-5fe17e7dc3d8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:54:54.232642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.211394 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816fb468-458b-4597-8da0-5fe17e7dc3d8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl\" (UID: \"816fb468-458b-4597-8da0-5fe17e7dc3d8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:54:54.232642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.219609 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl"] Oct 30 00:54:54.335109 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.335057 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz49g\" (UniqueName: \"kubernetes.io/projected/816fb468-458b-4597-8da0-5fe17e7dc3d8-kube-api-access-fz49g\") pod \"payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl\" (UID: \"816fb468-458b-4597-8da0-5fe17e7dc3d8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:54:54.335322 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.335122 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816fb468-458b-4597-8da0-5fe17e7dc3d8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl\" (UID: \"816fb468-458b-4597-8da0-5fe17e7dc3d8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:54:54.444372 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.444127 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz49g\" (UniqueName: \"kubernetes.io/projected/816fb468-458b-4597-8da0-5fe17e7dc3d8-kube-api-access-fz49g\") pod \"payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl\" (UID: \"816fb468-458b-4597-8da0-5fe17e7dc3d8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:54:54.446347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.446315 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816fb468-458b-4597-8da0-5fe17e7dc3d8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl\" (UID: \"816fb468-458b-4597-8da0-5fe17e7dc3d8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:54:54.485455 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.485412 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:54:54.558340 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.557296 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:54:54.857708 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.839930 4927 generic.go:356] "Generic (PLEG): container finished" podID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" containerID="bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54" exitCode=0 Oct 30 00:54:54.857708 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.840342 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" event={"ID":"f0fcc8c5-19f6-4599-8038-f367ab8828a5","Type":"ContainerDied","Data":"bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54"} Oct 30 00:54:54.857708 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.840448 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" event={"ID":"f0fcc8c5-19f6-4599-8038-f367ab8828a5","Type":"ContainerDied","Data":"30e5fdb5d2752b87feb3d1693c86a30916f1792d67976487b03ddf324bd161fb"} Oct 30 00:54:54.857708 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.840489 4927 scope.go:122] "RemoveContainer" containerID="bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54" Oct 30 00:54:54.857708 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:54.840872 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn" Oct 30 00:54:54.857708 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:54:54.847979 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc407565c_0ba4_4410_bedc_e6c9d4e5afe7.slice/crio-64affd99a046d8ae31b39ff1b76ec69868b169fbc3a87752bd21e12572f7751e WatchSource:0}: Error finding container 64affd99a046d8ae31b39ff1b76ec69868b169fbc3a87752bd21e12572f7751e: Status 404 returned error can't find the container with id 64affd99a046d8ae31b39ff1b76ec69868b169fbc3a87752bd21e12572f7751e Oct 30 00:54:55.113952 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.113910 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk"] Oct 30 00:54:55.157693 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.157647 4927 scope.go:122] "RemoveContainer" containerID="3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3" Oct 30 00:54:55.290897 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.290868 4927 scope.go:122] "RemoveContainer" containerID="bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54" Oct 30 00:54:55.291262 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:54:55.291234 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54\": container with ID starting with bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54 not found: ID does not exist" containerID="bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54" Oct 30 00:54:55.291340 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.291277 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54"} err="failed to get container status \"bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54\": rpc error: code = NotFound desc = could not find container \"bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54\": container with ID starting with bb0d3eecbf600c29c2c06132ea14b7429bf091d4eaa82da58f1f489479e0fc54 not found: ID does not exist" Oct 30 00:54:55.291340 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.291304 4927 scope.go:122] "RemoveContainer" containerID="3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3" Oct 30 00:54:55.291963 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:54:55.291638 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3\": container with ID starting with 3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3 not found: ID does not exist" containerID="3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3" Oct 30 00:54:55.291963 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.291712 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3"} err="failed to get container status \"3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3\": rpc error: code = NotFound desc = could not find container \"3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3\": container with ID starting with 3f110d408484b4fe0dec45bf752e7504ab866c03e46c42dc2a2efaa6bb9c54d3 not found: ID does not exist" Oct 30 00:54:55.312646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.311532 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn"] Oct 30 00:54:55.324614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.323655 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-573837e3-openshift-default-7779f764c-crrqn"] Oct 30 00:54:55.336067 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.336033 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl"] Oct 30 00:54:55.336537 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:54:55.336506 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816fb468_458b_4597_8da0_5fe17e7dc3d8.slice/crio-a0ca072e7d903898cf780b4c259198e1da0fa853997ec47b84578f0daaa98792 WatchSource:0}: Error finding container a0ca072e7d903898cf780b4c259198e1da0fa853997ec47b84578f0daaa98792: Status 404 returned error can't find the container with id a0ca072e7d903898cf780b4c259198e1da0fa853997ec47b84578f0daaa98792 Oct 30 00:54:55.574891 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.574810 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0fcc8c5-19f6-4599-8038-f367ab8828a5" path="/var/lib/kubelet/pods/f0fcc8c5-19f6-4599-8038-f367ab8828a5/volumes" Oct 30 00:54:55.851964 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.851856 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" event={"ID":"816fb468-458b-4597-8da0-5fe17e7dc3d8","Type":"ContainerStarted","Data":"113ee3b7c5940e75ef3bf634725b06eaa644fa67fc71893cb345ae4ce4565b01"} Oct 30 00:54:55.851964 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.851926 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" event={"ID":"816fb468-458b-4597-8da0-5fe17e7dc3d8","Type":"ContainerStarted","Data":"a0ca072e7d903898cf780b4c259198e1da0fa853997ec47b84578f0daaa98792"} Oct 30 00:54:55.851964 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.851956 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:54:55.853581 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.853543 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" event={"ID":"c407565c-0ba4-4410-bedc-e6c9d4e5afe7","Type":"ContainerStarted","Data":"b206851fdd39536d3189975be76cf7bc1ce01e075e965fcf1e049ea26a52fb76"} Oct 30 00:54:55.853659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.853610 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" event={"ID":"c407565c-0ba4-4410-bedc-e6c9d4e5afe7","Type":"ContainerStarted","Data":"64affd99a046d8ae31b39ff1b76ec69868b169fbc3a87752bd21e12572f7751e"} Oct 30 00:54:55.853700 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.853669 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:54:55.891914 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.891856 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" podStartSLOduration=2.891836986 podStartE2EDuration="2.891836986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:55.890247802 +0000 UTC m=+1234.462174020" watchObservedRunningTime="2026-10-30 00:54:55.891836986 +0000 UTC m=+1234.463763253" Oct 30 00:54:55.937227 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:55.937169 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" podStartSLOduration=2.937154273 podStartE2EDuration="2.937154273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:55.932906037 +0000 UTC m=+1234.504832240" watchObservedRunningTime="2026-10-30 00:54:55.937154273 +0000 UTC m=+1234.509080494" Oct 30 00:54:56.198393 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.198349 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v"] Oct 30 00:54:56.204801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.204777 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:54:56.218936 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.218905 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-95492e\"" Oct 30 00:54:56.219097 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.219042 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-95492e-dockercfg-z7nhf\"" Oct 30 00:54:56.224157 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.224125 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v"] Oct 30 00:54:56.248375 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.248339 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w"] Oct 30 00:54:56.254543 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.254512 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:54:56.273459 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.273424 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w"] Oct 30 00:54:56.304000 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.303965 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff872648-3c7b-4b11-90b1-8d34939b17ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v\" (UID: \"ff872648-3c7b-4b11-90b1-8d34939b17ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:54:56.304200 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.304109 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4bc\" (UniqueName: \"kubernetes.io/projected/ff872648-3c7b-4b11-90b1-8d34939b17ce-kube-api-access-tn4bc\") pod \"payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v\" (UID: \"ff872648-3c7b-4b11-90b1-8d34939b17ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:54:56.469667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.468676 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4bc\" (UniqueName: \"kubernetes.io/projected/ff872648-3c7b-4b11-90b1-8d34939b17ce-kube-api-access-tn4bc\") pod \"payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v\" (UID: \"ff872648-3c7b-4b11-90b1-8d34939b17ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:54:56.469667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.468906 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651c823f-662b-404a-9374-c0f641f322df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w\" (UID: \"651c823f-662b-404a-9374-c0f641f322df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:54:56.469667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.469100 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff872648-3c7b-4b11-90b1-8d34939b17ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v\" (UID: \"ff872648-3c7b-4b11-90b1-8d34939b17ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:54:56.469667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.469176 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7s77\" (UniqueName: \"kubernetes.io/projected/651c823f-662b-404a-9374-c0f641f322df-kube-api-access-b7s77\") pod \"payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w\" (UID: \"651c823f-662b-404a-9374-c0f641f322df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:54:56.494599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.473095 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff872648-3c7b-4b11-90b1-8d34939b17ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v\" (UID: \"ff872648-3c7b-4b11-90b1-8d34939b17ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:54:56.573939 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.573895 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651c823f-662b-404a-9374-c0f641f322df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w\" (UID: \"651c823f-662b-404a-9374-c0f641f322df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:54:56.574653 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.574622 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7s77\" (UniqueName: \"kubernetes.io/projected/651c823f-662b-404a-9374-c0f641f322df-kube-api-access-b7s77\") pod \"payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w\" (UID: \"651c823f-662b-404a-9374-c0f641f322df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:54:56.576354 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.576329 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651c823f-662b-404a-9374-c0f641f322df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w\" (UID: \"651c823f-662b-404a-9374-c0f641f322df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:54:56.606483 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.606448 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4bc\" (UniqueName: \"kubernetes.io/projected/ff872648-3c7b-4b11-90b1-8d34939b17ce-kube-api-access-tn4bc\") pod \"payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v\" (UID: \"ff872648-3c7b-4b11-90b1-8d34939b17ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:54:56.614812 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.607679 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7s77\" (UniqueName: \"kubernetes.io/projected/651c823f-662b-404a-9374-c0f641f322df-kube-api-access-b7s77\") pod \"payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w\" (UID: \"651c823f-662b-404a-9374-c0f641f322df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:54:56.824481 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.824394 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:54:56.882166 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:56.882128 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:54:57.150464 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.150431 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v"] Oct 30 00:54:57.150861 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:54:57.150492 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff872648_3c7b_4b11_90b1_8d34939b17ce.slice/crio-a3fb885c99b955ba789cab59c3810d4956db84f11294301e992747cc28d9a7e7 WatchSource:0}: Error finding container a3fb885c99b955ba789cab59c3810d4956db84f11294301e992747cc28d9a7e7: Status 404 returned error can't find the container with id a3fb885c99b955ba789cab59c3810d4956db84f11294301e992747cc28d9a7e7 Oct 30 00:54:57.325370 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.325325 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w"] Oct 30 00:54:57.887971 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.887927 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" event={"ID":"651c823f-662b-404a-9374-c0f641f322df","Type":"ContainerStarted","Data":"f1feecee5b81e8bdf6a7c5577a64235bffd47eb685ee2afc68fddf36812cc655"} Oct 30 00:54:57.888194 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.888025 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" event={"ID":"651c823f-662b-404a-9374-c0f641f322df","Type":"ContainerStarted","Data":"b979e76e91c020b55b8ea924369173f9ebdc1ec7867d4e8203c6b383481cc5ba"} Oct 30 00:54:57.888249 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.888192 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:54:57.890072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.890045 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" event={"ID":"ff872648-3c7b-4b11-90b1-8d34939b17ce","Type":"ContainerStarted","Data":"a2e19378d9e052fb44629c34170b01047012a84f959b6ba676d291ed2cc0af39"} Oct 30 00:54:57.890635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.890085 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" event={"ID":"ff872648-3c7b-4b11-90b1-8d34939b17ce","Type":"ContainerStarted","Data":"a3fb885c99b955ba789cab59c3810d4956db84f11294301e992747cc28d9a7e7"} Oct 30 00:54:57.890635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.890149 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:54:57.928751 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.928694 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" podStartSLOduration=1.928680018 podStartE2EDuration="1.928680018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:57.926139206 +0000 UTC m=+1236.498065466" watchObservedRunningTime="2026-10-30 00:54:57.928680018 +0000 UTC m=+1236.500606240" Oct 30 00:54:57.967050 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:54:57.966985 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" podStartSLOduration=1.966965118 podStartE2EDuration="1.966965118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:57.964541961 +0000 UTC m=+1236.536468183" watchObservedRunningTime="2026-10-30 00:54:57.966965118 +0000 UTC m=+1236.538891344" Oct 30 00:55:04.447145 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:55:04.447099 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:55:06.899597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:06.896909 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 00:55:06.904648 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:06.902418 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 00:55:08.901888 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:08.901857 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 00:55:08.902383 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:08.901937 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 00:55:11.584930 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.584895 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5"] Oct 30 00:55:11.590763 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.590734 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.619833 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.619801 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-bea3ef-serving-cert\"" Oct 30 00:55:11.635844 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.635806 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5"] Oct 30 00:55:11.712722 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.712685 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.712910 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.712727 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdn7\" (UniqueName: \"kubernetes.io/projected/79b8fa70-f329-4ae1-8384-068f7fd1d151-kube-api-access-qqdn7\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.712910 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.712872 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.712992 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.712979 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8fa70-f329-4ae1-8384-068f7fd1d151-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.814706 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.814668 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.815532 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.815497 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdn7\" (UniqueName: \"kubernetes.io/projected/79b8fa70-f329-4ae1-8384-068f7fd1d151-kube-api-access-qqdn7\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.815894 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.815867 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.816228 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.816207 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8fa70-f329-4ae1-8384-068f7fd1d151-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.817443 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.817055 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8fa70-f329-4ae1-8384-068f7fd1d151-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.820488 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.820455 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.821710 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.820666 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.910704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.901475 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdn7\" (UniqueName: \"kubernetes.io/projected/79b8fa70-f329-4ae1-8384-068f7fd1d151-kube-api-access-qqdn7\") pod \"maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:11.910704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:11.910626 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:12.343745 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:12.343699 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5"] Oct 30 00:55:12.343884 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:55:12.343859 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b8fa70_f329_4ae1_8384_068f7fd1d151.slice/crio-0fd927cdfb17a54d985b8e5b86795a17ba5d9a78112109038c83c9488eb9c800 WatchSource:0}: Error finding container 0fd927cdfb17a54d985b8e5b86795a17ba5d9a78112109038c83c9488eb9c800: Status 404 returned error can't find the container with id 0fd927cdfb17a54d985b8e5b86795a17ba5d9a78112109038c83c9488eb9c800 Oct 30 00:55:12.372916 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:12.372878 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" event={"ID":"79b8fa70-f329-4ae1-8384-068f7fd1d151","Type":"ContainerStarted","Data":"0fd927cdfb17a54d985b8e5b86795a17ba5d9a78112109038c83c9488eb9c800"} Oct 30 00:55:13.462598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:13.462089 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" event={"ID":"79b8fa70-f329-4ae1-8384-068f7fd1d151","Type":"ContainerStarted","Data":"c90f075f13be45740bd6b95272508185ebc402d8a5ffc85e59516cf32120ddfa"} Oct 30 00:55:13.463096 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:13.462780 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:13.618957 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:13.618891 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" podStartSLOduration=2.265431791 podStartE2EDuration="2.618869534s" totalImagesPullingTime="353.437743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:13.52388975 +0000 UTC m=+1252.095815973" watchObservedRunningTime="2026-10-30 00:55:13.618869534 +0000 UTC m=+1252.190795758" Oct 30 00:55:15.677305 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.677267 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp"] Oct 30 00:55:15.682549 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.682524 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.703996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.703961 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp"] Oct 30 00:55:15.707250 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.707220 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-95492e-serving-cert\"" Oct 30 00:55:15.803658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.803615 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.803881 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.803753 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.803881 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.803871 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f27fb\" (UniqueName: \"kubernetes.io/projected/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-kube-api-access-f27fb\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.803999 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.803957 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.905536 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.905466 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.906916 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.906778 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.907402 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.907374 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.908063 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.908039 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.909617 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.909589 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27fb\" (UniqueName: \"kubernetes.io/projected/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-kube-api-access-f27fb\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.910330 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:55:15.908233 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-serving-cert: secret "maas-api-e2e-shared-a-w1-95492e-serving-cert" not found Oct 30 00:55:15.911707 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:55:15.910530 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls podName:f3fbc92f-3abd-4859-a96c-70d5c6e7078c nodeName:}" failed. No retries permitted until 2026-10-30 00:55:16.410500828 +0000 UTC m=+1254.982427051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls") pod "maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" (UID: "f3fbc92f-3abd-4859-a96c-70d5c6e7078c") : secret "maas-api-e2e-shared-a-w1-95492e-serving-cert" not found Oct 30 00:55:15.914947 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.914806 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:15.958635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:15.958008 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f27fb\" (UniqueName: \"kubernetes.io/projected/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-kube-api-access-f27fb\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:16.429754 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:16.429696 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:16.437500 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:16.437460 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:16.616500 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:16.603946 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:17.048622 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:17.043503 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp"] Oct 30 00:55:17.512755 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:17.512713 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" event={"ID":"f3fbc92f-3abd-4859-a96c-70d5c6e7078c","Type":"ContainerStarted","Data":"a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd"} Oct 30 00:55:17.513084 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:17.513059 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" event={"ID":"f3fbc92f-3abd-4859-a96c-70d5c6e7078c","Type":"ContainerStarted","Data":"25c55d9bbbb6bfd68e966e5e740c0bff3068bc0c0bff3d3dfdbd4f4a35283110"} Oct 30 00:55:17.514357 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:17.514333 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:17.561892 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:17.561813 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" podStartSLOduration=2.196158397 podStartE2EDuration="2.56179349s" totalImagesPullingTime="365.635093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:17.553841566 +0000 UTC m=+1256.125767790" watchObservedRunningTime="2026-10-30 00:55:17.56179349 +0000 UTC m=+1256.133719713" Oct 30 00:55:19.476608 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:19.476559 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 00:55:23.565725 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:23.560835 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 00:55:24.516595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:24.515685 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 00:55:33.842560 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:33.842524 4927 generic.go:356] "Generic (PLEG): container finished" podID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerID="11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f" exitCode=0 Oct 30 00:55:33.843169 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:33.842605 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" event={"ID":"43321879-3683-44ec-b6b6-11820f4ad9f2","Type":"ContainerDied","Data":"11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f"} Oct 30 00:55:33.843236 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:33.843212 4927 scope.go:122] "RemoveContainer" containerID="11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f" Oct 30 00:55:33.845224 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:33.845184 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:33.845350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:33.845282 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:33.845350 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:33.845336 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:34.448078 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:55:34.448034 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:55:34.920599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:34.911384 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" event={"ID":"43321879-3683-44ec-b6b6-11820f4ad9f2","Type":"ContainerStarted","Data":"291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855"} Oct 30 00:55:35.417085 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:35.417046 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:55:35.418242 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:35.418214 4927 patch_prober.go:29] interesting pod/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:55:35.418381 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:35.418277 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:55:36.431301 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:36.431255 4927 patch_prober.go:29] interesting pod/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:55:36.438835 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:36.438771 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:55:37.420018 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:37.419977 4927 patch_prober.go:29] interesting pod/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:55:37.420344 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:37.420314 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:55:38.417466 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:38.417430 4927 patch_prober.go:29] interesting pod/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:55:38.417948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:38.417487 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:55:39.416958 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:39.416916 4927 patch_prober.go:29] interesting pod/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:55:39.417166 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:39.416974 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:55:39.904914 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:39.904875 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:55:40.432594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:40.420530 4927 patch_prober.go:29] interesting pod/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 00:55:40.432594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:40.420613 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 00:55:41.421785 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:41.421747 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:55:41.423330 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:41.423293 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 00:55:51.019848 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.019809 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q"] Oct 30 00:55:51.026631 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.026598 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.050838 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.050808 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-16c1bb-openshift-default-dockercfg-h52zt\"" Oct 30 00:55:51.051562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.051529 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q"] Oct 30 00:55:51.209630 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.209160 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bc5q\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-kube-api-access-4bc5q\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.210158 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.210052 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.210432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.210392 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.210741 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.210625 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.210983 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.210957 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.211124 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.211107 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee87ca52-c410-4f3b-9906-ff823188c656-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.211336 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.211319 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.211639 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.211600 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee87ca52-c410-4f3b-9906-ff823188c656-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.212067 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.212000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.354297 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.354192 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bc5q\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-kube-api-access-4bc5q\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.354653 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.354623 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.354847 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.354827 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.355155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.355128 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.355309 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.355289 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.355452 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.355433 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee87ca52-c410-4f3b-9906-ff823188c656-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.355623 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.355604 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.355908 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.355884 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee87ca52-c410-4f3b-9906-ff823188c656-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.356256 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.356233 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.366832 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.357051 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.367072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.358784 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.367654 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.364974 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.367850 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.366057 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee87ca52-c410-4f3b-9906-ff823188c656-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.367981 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.366283 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.375700 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.375664 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.383629 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.383596 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee87ca52-c410-4f3b-9906-ff823188c656-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.408131 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.408100 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bc5q\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-kube-api-access-4bc5q\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.415031 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.414591 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:51.659171 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:51.659127 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:52.447642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:52.447608 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q"] Oct 30 00:55:52.449179 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:55:52.448918 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee87ca52_c410_4f3b_9906_ff823188c656.slice/crio-34826da47928c1e2d70d698da97f5fdc89ca54969f2893fc23fad8526e32c77a WatchSource:0}: Error finding container 34826da47928c1e2d70d698da97f5fdc89ca54969f2893fc23fad8526e32c77a: Status 404 returned error can't find the container with id 34826da47928c1e2d70d698da97f5fdc89ca54969f2893fc23fad8526e32c77a Oct 30 00:55:52.451612 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:52.451561 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:52.451695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:52.451660 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:52.451734 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:52.451692 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:53.165621 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:53.161773 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" event={"ID":"ee87ca52-c410-4f3b-9906-ff823188c656","Type":"ContainerStarted","Data":"836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d"} Oct 30 00:55:53.165621 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:53.162088 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" event={"ID":"ee87ca52-c410-4f3b-9906-ff823188c656","Type":"ContainerStarted","Data":"34826da47928c1e2d70d698da97f5fdc89ca54969f2893fc23fad8526e32c77a"} Oct 30 00:55:53.442474 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:53.442317 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podStartSLOduration=3.442271367 podStartE2EDuration="3.442271367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:53.24304584 +0000 UTC m=+1291.814972064" watchObservedRunningTime="2026-10-30 00:55:53.442271367 +0000 UTC m=+1292.014197589" Oct 30 00:55:53.700689 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:53.700593 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 00:55:53.701363 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:53.701329 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 00:55:53.856606 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:53.855787 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:54.665803 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:54.665771 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:55.324676 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:55.296175 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:55:55.351603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:55:55.344895 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 00:56:04.447770 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:56:04.447678 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:56:11.218029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.217989 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh"] Oct 30 00:56:11.223349 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.223321 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:11.242304 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.242273 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh"] Oct 30 00:56:11.247532 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.247493 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a4fe0143\"" Oct 30 00:56:11.247532 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.247500 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a4fe0143-dockercfg-x4799\"" Oct 30 00:56:11.438413 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.433754 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6"] Oct 30 00:56:11.498846 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.498411 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8616c4e2-8249-435d-a67e-4703af0cadf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh\" (UID: \"8616c4e2-8249-435d-a67e-4703af0cadf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:11.498846 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.498523 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5lkr\" (UniqueName: \"kubernetes.io/projected/8616c4e2-8249-435d-a67e-4703af0cadf1-kube-api-access-r5lkr\") pod \"payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh\" (UID: \"8616c4e2-8249-435d-a67e-4703af0cadf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:11.653626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.609943 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8616c4e2-8249-435d-a67e-4703af0cadf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh\" (UID: \"8616c4e2-8249-435d-a67e-4703af0cadf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:11.660167 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.654016 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5lkr\" (UniqueName: \"kubernetes.io/projected/8616c4e2-8249-435d-a67e-4703af0cadf1-kube-api-access-r5lkr\") pod \"payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh\" (UID: \"8616c4e2-8249-435d-a67e-4703af0cadf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:11.660624 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.611074 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8616c4e2-8249-435d-a67e-4703af0cadf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh\" (UID: \"8616c4e2-8249-435d-a67e-4703af0cadf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:11.751833 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.751735 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:11.782812 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.782779 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5lkr\" (UniqueName: \"kubernetes.io/projected/8616c4e2-8249-435d-a67e-4703af0cadf1-kube-api-access-r5lkr\") pod \"payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh\" (UID: \"8616c4e2-8249-435d-a67e-4703af0cadf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:11.786828 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.785219 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6"] Oct 30 00:56:11.843216 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.843175 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:11.889634 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.867499 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25r4g\" (UniqueName: \"kubernetes.io/projected/b1b442ce-b4dd-4d11-bfce-de988c90e482-kube-api-access-25r4g\") pod \"payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6\" (UID: \"b1b442ce-b4dd-4d11-bfce-de988c90e482\") " pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:11.890484 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.890443 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b442ce-b4dd-4d11-bfce-de988c90e482-plugins-config-volume\") pod \"payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6\" (UID: \"b1b442ce-b4dd-4d11-bfce-de988c90e482\") " pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:11.995497 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.993817 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25r4g\" (UniqueName: \"kubernetes.io/projected/b1b442ce-b4dd-4d11-bfce-de988c90e482-kube-api-access-25r4g\") pod \"payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6\" (UID: \"b1b442ce-b4dd-4d11-bfce-de988c90e482\") " pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:11.995497 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.994186 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b442ce-b4dd-4d11-bfce-de988c90e482-plugins-config-volume\") pod \"payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6\" (UID: \"b1b442ce-b4dd-4d11-bfce-de988c90e482\") " pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:11.995497 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:11.995281 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b442ce-b4dd-4d11-bfce-de988c90e482-plugins-config-volume\") pod \"payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6\" (UID: \"b1b442ce-b4dd-4d11-bfce-de988c90e482\") " pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:12.023987 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.022204 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25r4g\" (UniqueName: \"kubernetes.io/projected/b1b442ce-b4dd-4d11-bfce-de988c90e482-kube-api-access-25r4g\") pod \"payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6\" (UID: \"b1b442ce-b4dd-4d11-bfce-de988c90e482\") " pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:12.228829 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.228794 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:12.255276 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.255190 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh"] Oct 30 00:56:12.557913 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:56:12.557881 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b442ce_b4dd_4d11_bfce_de988c90e482.slice/crio-1fef3492cd6965200d33bd770bcddb3ea4d2dc23f9288c345e0b57017632b266 WatchSource:0}: Error finding container 1fef3492cd6965200d33bd770bcddb3ea4d2dc23f9288c345e0b57017632b266: Status 404 returned error can't find the container with id 1fef3492cd6965200d33bd770bcddb3ea4d2dc23f9288c345e0b57017632b266 Oct 30 00:56:12.558165 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.558133 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6"] Oct 30 00:56:12.938363 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.938233 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" event={"ID":"b1b442ce-b4dd-4d11-bfce-de988c90e482","Type":"ContainerStarted","Data":"5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625"} Oct 30 00:56:12.938778 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.938659 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" event={"ID":"b1b442ce-b4dd-4d11-bfce-de988c90e482","Type":"ContainerStarted","Data":"1fef3492cd6965200d33bd770bcddb3ea4d2dc23f9288c345e0b57017632b266"} Oct 30 00:56:12.938948 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.938933 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:12.944729 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.944695 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" event={"ID":"8616c4e2-8249-435d-a67e-4703af0cadf1","Type":"ContainerStarted","Data":"a8805b77d35f91be3d476c62585f80389001d6e5e65300397bd0fbbac8b7de0e"} Oct 30 00:56:12.944942 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.944925 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" event={"ID":"8616c4e2-8249-435d-a67e-4703af0cadf1","Type":"ContainerStarted","Data":"b66d4b957982a636e1ec3612d6b92067e3572d711aa3ae52b1bc52ec4e55111d"} Oct 30 00:56:12.945712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:12.945645 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:13.001636 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:13.001550 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" podStartSLOduration=2.0015315 podStartE2EDuration="2.0015315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:13.001009078 +0000 UTC m=+1311.572935300" watchObservedRunningTime="2026-10-30 00:56:13.0015315 +0000 UTC m=+1311.573457721" Oct 30 00:56:13.077601 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:13.076290 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" podStartSLOduration=2.076267885 podStartE2EDuration="2.076267885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:13.042093833 +0000 UTC m=+1311.614020056" watchObservedRunningTime="2026-10-30 00:56:13.076267885 +0000 UTC m=+1311.648194108" Oct 30 00:56:23.957558 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:23.957513 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:56:23.958119 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:23.957975 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:56:34.446272 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:56:34.446228 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:56:45.573623 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:45.573566 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8979-c98f1b"] Oct 30 00:56:45.579891 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:45.579864 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8979-c98f1b" Oct 30 00:56:45.585249 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:45.585219 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8979-c98f1b"] Oct 30 00:56:45.593945 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:45.593920 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v4hbx\"" Oct 30 00:56:45.712530 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:45.712488 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxs28\" (UniqueName: \"kubernetes.io/projected/f75546c4-b1da-446a-9aed-85cf2f7fe3d7-kube-api-access-xxs28\") pod \"test-curl-8979-c98f1b\" (UID: \"f75546c4-b1da-446a-9aed-85cf2f7fe3d7\") " pod="opendatahub/test-curl-8979-c98f1b" Oct 30 00:56:45.814951 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:45.814911 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxs28\" (UniqueName: \"kubernetes.io/projected/f75546c4-b1da-446a-9aed-85cf2f7fe3d7-kube-api-access-xxs28\") pod \"test-curl-8979-c98f1b\" (UID: \"f75546c4-b1da-446a-9aed-85cf2f7fe3d7\") " pod="opendatahub/test-curl-8979-c98f1b" Oct 30 00:56:45.837720 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:45.837599 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxs28\" (UniqueName: \"kubernetes.io/projected/f75546c4-b1da-446a-9aed-85cf2f7fe3d7-kube-api-access-xxs28\") pod \"test-curl-8979-c98f1b\" (UID: \"f75546c4-b1da-446a-9aed-85cf2f7fe3d7\") " pod="opendatahub/test-curl-8979-c98f1b" Oct 30 00:56:45.898806 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:45.898770 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8979-c98f1b" Oct 30 00:56:46.566839 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:56:46.566786 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf75546c4_b1da_446a_9aed_85cf2f7fe3d7.slice/crio-23f99a02a3c3ad3a780753d95ac73037c0331d78364d74493476b1be05c7c59f WatchSource:0}: Error finding container 23f99a02a3c3ad3a780753d95ac73037c0331d78364d74493476b1be05c7c59f: Status 404 returned error can't find the container with id 23f99a02a3c3ad3a780753d95ac73037c0331d78364d74493476b1be05c7c59f Oct 30 00:56:46.630588 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:46.630534 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8979-c98f1b"] Oct 30 00:56:47.595806 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:47.595767 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8979-c98f1b" event={"ID":"f75546c4-b1da-446a-9aed-85cf2f7fe3d7","Type":"ContainerStarted","Data":"da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539"} Oct 30 00:56:47.595806 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:47.595816 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8979-c98f1b" event={"ID":"f75546c4-b1da-446a-9aed-85cf2f7fe3d7","Type":"ContainerStarted","Data":"23f99a02a3c3ad3a780753d95ac73037c0331d78364d74493476b1be05c7c59f"} Oct 30 00:56:47.634860 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:47.634794 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8979-c98f1b" podStartSLOduration=2.634779037 podStartE2EDuration="2.634779037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:47.632604229 +0000 UTC m=+1346.204530451" watchObservedRunningTime="2026-10-30 00:56:47.634779037 +0000 UTC m=+1346.206705258" Oct 30 00:56:48.504111 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:48.504074 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8979-c98f1b"] Oct 30 00:56:48.511023 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:48.510983 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8979-c98f1b"] Oct 30 00:56:49.613914 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:49.613866 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8979-c98f1b" podUID="f75546c4-b1da-446a-9aed-85cf2f7fe3d7" containerName="test-curl-8979-c98f1b" containerID="cri-o://da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539" gracePeriod=2 Oct 30 00:56:51.910045 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:51.910009 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8979-c98f1b" Oct 30 00:56:52.021229 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:52.021190 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f75546c4-b1da-446a-9aed-85cf2f7fe3d7-kube-api-access-xxs28\" (UniqueName: \"kubernetes.io/projected/f75546c4-b1da-446a-9aed-85cf2f7fe3d7-kube-api-access-xxs28\") pod \"f75546c4-b1da-446a-9aed-85cf2f7fe3d7\" (UID: \"f75546c4-b1da-446a-9aed-85cf2f7fe3d7\") " Oct 30 00:56:52.023537 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:52.023503 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75546c4-b1da-446a-9aed-85cf2f7fe3d7-kube-api-access-xxs28" pod "f75546c4-b1da-446a-9aed-85cf2f7fe3d7" (UID: "f75546c4-b1da-446a-9aed-85cf2f7fe3d7"). InnerVolumeSpecName "kube-api-access-xxs28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:52.143687 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:52.124971 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxs28\" (UniqueName: \"kubernetes.io/projected/f75546c4-b1da-446a-9aed-85cf2f7fe3d7-kube-api-access-xxs28\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:56:52.651956 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:52.651918 4927 generic.go:356] "Generic (PLEG): container finished" podID="f75546c4-b1da-446a-9aed-85cf2f7fe3d7" containerID="da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539" exitCode=137 Oct 30 00:56:52.652143 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:52.652030 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8979-c98f1b" Oct 30 00:56:52.652143 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:52.652056 4927 scope.go:122] "RemoveContainer" containerID="da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539" Oct 30 00:56:52.667683 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:52.667656 4927 scope.go:122] "RemoveContainer" containerID="da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539" Oct 30 00:56:52.668041 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:56:52.668017 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539\": container with ID starting with da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539 not found: ID does not exist" containerID="da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539" Oct 30 00:56:52.668155 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:52.668046 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539"} err="failed to get container status \"da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539\": rpc error: code = NotFound desc = could not find container \"da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539\": container with ID starting with da11ac99c5d3313d9fc781c0feddc24b1f8970c1b8de416d997cde87af973539 not found: ID does not exist" Oct 30 00:56:52.678404 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:56:52.678370 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8979-c98f1b\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="f75546c4-b1da-446a-9aed-85cf2f7fe3d7" pod="opendatahub/test-curl-8979-c98f1b" Oct 30 00:56:52.728138 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:56:52.728100 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf75546c4_b1da_446a_9aed_85cf2f7fe3d7.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:56:53.593198 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:56:53.593159 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f75546c4-b1da-446a-9aed-85cf2f7fe3d7" path="/var/lib/kubelet/pods/f75546c4-b1da-446a-9aed-85cf2f7fe3d7/volumes" Oct 30 00:57:04.602604 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:57:04.597771 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:57:34.445854 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:57:34.445752 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:57:50.468662 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:50.468621 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk"] Oct 30 00:57:50.469270 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:50.468850 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" containerID="cri-o://e7f04844e1d2d39ab1931c2efb4b792e1bd889f47f601721509818ae0135dc60" gracePeriod=30 Oct 30 00:57:50.480564 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:50.480529 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb"] Oct 30 00:57:50.480865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:50.480835 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" containerID="cri-o://9d784856571772591add28df8e0b90b090c921d7c694fb0895b1b5f430dacd5c" gracePeriod=30 Oct 30 00:57:50.488070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:50.488038 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z"] Oct 30 00:57:50.488291 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:50.488266 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" containerID="cri-o://da5e491af6571aa496e21a52b86556f7ccaa99788fe288a24a30d0757cf8a7ec" gracePeriod=30 Oct 30 00:57:51.447692 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:51.447649 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:57:51.448014 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:51.447979 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:57:56.673975 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:56.673749 4927 generic.go:356] "Generic (PLEG): container finished" podID="6c47f328-0846-4b77-b15e-d4561701475f" containerID="e7f04844e1d2d39ab1931c2efb4b792e1bd889f47f601721509818ae0135dc60" exitCode=0 Oct 30 00:57:56.673975 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:56.673868 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" event={"ID":"6c47f328-0846-4b77-b15e-d4561701475f","Type":"ContainerDied","Data":"e7f04844e1d2d39ab1931c2efb4b792e1bd889f47f601721509818ae0135dc60"} Oct 30 00:57:56.888735 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:56.843216 4927 generic.go:356] "Generic (PLEG): container finished" podID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerID="9d784856571772591add28df8e0b90b090c921d7c694fb0895b1b5f430dacd5c" exitCode=0 Oct 30 00:57:56.888735 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:56.843457 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" event={"ID":"3b8008c2-15fb-4bf0-80e2-575a4f60abe6","Type":"ContainerDied","Data":"9d784856571772591add28df8e0b90b090c921d7c694fb0895b1b5f430dacd5c"} Oct 30 00:57:57.327593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.326876 4927 generic.go:356] "Generic (PLEG): container finished" podID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerID="da5e491af6571aa496e21a52b86556f7ccaa99788fe288a24a30d0757cf8a7ec" exitCode=0 Oct 30 00:57:57.327593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.327068 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" event={"ID":"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5","Type":"ContainerDied","Data":"da5e491af6571aa496e21a52b86556f7ccaa99788fe288a24a30d0757cf8a7ec"} Oct 30 00:57:57.330680 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.330603 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:57:57.373193 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.373158 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:57:57.380547 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.380513 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:57:57.459210 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459167 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-data\") pod \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " Oct 30 00:57:57.459425 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459250 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-envoy\") pod \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " Oct 30 00:57:57.459425 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459301 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-certs\") pod \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " Oct 30 00:57:57.459425 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459331 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-data\") pod \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " Oct 30 00:57:57.459425 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459393 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-socket\") pod \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " Oct 30 00:57:57.459672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459509 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-socket\") pod \"6c47f328-0846-4b77-b15e-d4561701475f\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " Oct 30 00:57:57.459672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459553 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-token\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-token\") pod \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " Oct 30 00:57:57.459672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459617 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-kube-api-access-nnzx2\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-kube-api-access-nnzx2\") pod \"6c47f328-0846-4b77-b15e-d4561701475f\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " Oct 30 00:57:57.459851 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459669 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-data\") pod \"6c47f328-0846-4b77-b15e-d4561701475f\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " Oct 30 00:57:57.459851 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459707 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-credential-socket\") pod \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " Oct 30 00:57:57.459851 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459752 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-podinfo\") pod \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " Oct 30 00:57:57.459851 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459784 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-socket\") pod \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " Oct 30 00:57:57.460051 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459863 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-istio-token\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-istio-token\") pod \"6c47f328-0846-4b77-b15e-d4561701475f\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " Oct 30 00:57:57.460051 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459908 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c47f328-0846-4b77-b15e-d4561701475f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c47f328-0846-4b77-b15e-d4561701475f-istiod-ca-cert\") pod \"6c47f328-0846-4b77-b15e-d4561701475f\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " Oct 30 00:57:57.460051 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.459949 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-podinfo\") pod \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " Oct 30 00:57:57.460203 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460068 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istiod-ca-cert\") pod \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " Oct 30 00:57:57.460203 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460105 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-envoy\") pod \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " Oct 30 00:57:57.460203 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460140 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-token\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-token\") pod \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " Oct 30 00:57:57.460203 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460187 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istiod-ca-cert\") pod \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " Oct 30 00:57:57.460404 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460221 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-kube-api-access-rg8tc\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-kube-api-access-rg8tc\") pod \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\" (UID: \"3b8008c2-15fb-4bf0-80e2-575a4f60abe6\") " Oct 30 00:57:57.460404 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460254 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-kube-api-access-rzkwv\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-kube-api-access-rzkwv\") pod \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " Oct 30 00:57:57.460404 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460316 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-certs\") pod \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " Oct 30 00:57:57.460555 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460468 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-envoy\") pod \"6c47f328-0846-4b77-b15e-d4561701475f\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " Oct 30 00:57:57.460555 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460508 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c47f328-0846-4b77-b15e-d4561701475f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c47f328-0846-4b77-b15e-d4561701475f-istio-podinfo\") pod \"6c47f328-0846-4b77-b15e-d4561701475f\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " Oct 30 00:57:57.460555 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460546 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-credential-socket\") pod \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\" (UID: \"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5\") " Oct 30 00:57:57.460726 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460679 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-credential-socket\") pod \"6c47f328-0846-4b77-b15e-d4561701475f\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " Oct 30 00:57:57.460781 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.460753 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-certs\") pod \"6c47f328-0846-4b77-b15e-d4561701475f\" (UID: \"6c47f328-0846-4b77-b15e-d4561701475f\") " Oct 30 00:57:57.469029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.467925 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-certs" pod "6c47f328-0846-4b77-b15e-d4561701475f" (UID: "6c47f328-0846-4b77-b15e-d4561701475f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.469029 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.468308 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c47f328-0846-4b77-b15e-d4561701475f-istiod-ca-cert" pod "6c47f328-0846-4b77-b15e-d4561701475f" (UID: "6c47f328-0846-4b77-b15e-d4561701475f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:57.512348 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.510228 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt"] Oct 30 00:57:57.512348 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.510738 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-data" pod "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" (UID: "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.512348 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512322 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f75546c4-b1da-446a-9aed-85cf2f7fe3d7" containerName="test-curl-8979-c98f1b" Oct 30 00:57:57.512348 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512345 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f75546c4-b1da-446a-9aed-85cf2f7fe3d7" containerName="test-curl-8979-c98f1b" Oct 30 00:57:57.512709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512380 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" Oct 30 00:57:57.512709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512390 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" Oct 30 00:57:57.512709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512407 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" Oct 30 00:57:57.512709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512414 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" Oct 30 00:57:57.512709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512428 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" Oct 30 00:57:57.512709 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512436 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" Oct 30 00:57:57.512998 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512955 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" containerName="istio-proxy" Oct 30 00:57:57.512998 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.512975 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c47f328-0846-4b77-b15e-d4561701475f" containerName="istio-proxy" Oct 30 00:57:57.513089 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.513011 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f75546c4-b1da-446a-9aed-85cf2f7fe3d7" containerName="test-curl-8979-c98f1b" Oct 30 00:57:57.513089 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.513026 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" containerName="istio-proxy" Oct 30 00:57:57.519187 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.519012 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-certs" pod "3b8008c2-15fb-4bf0-80e2-575a4f60abe6" (UID: "3b8008c2-15fb-4bf0-80e2-575a4f60abe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.519356 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.519274 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-data" pod "3b8008c2-15fb-4bf0-80e2-575a4f60abe6" (UID: "3b8008c2-15fb-4bf0-80e2-575a4f60abe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.519738 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.519491 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-socket" pod "3b8008c2-15fb-4bf0-80e2-575a4f60abe6" (UID: "3b8008c2-15fb-4bf0-80e2-575a4f60abe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.519738 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.519717 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-socket" pod "6c47f328-0846-4b77-b15e-d4561701475f" (UID: "6c47f328-0846-4b77-b15e-d4561701475f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.522735 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.520646 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istiod-ca-cert" pod "3b8008c2-15fb-4bf0-80e2-575a4f60abe6" (UID: "3b8008c2-15fb-4bf0-80e2-575a4f60abe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:57.522735 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.522085 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.545804 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.541718 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-credential-socket" pod "3b8008c2-15fb-4bf0-80e2-575a4f60abe6" (UID: "3b8008c2-15fb-4bf0-80e2-575a4f60abe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571191 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571231 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571247 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c47f328-0846-4b77-b15e-d4561701475f-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571262 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571277 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571291 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571304 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571316 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571328 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.573313 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.571633 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-certs" pod "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" (UID: "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.579316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.579257 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-socket" pod "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" (UID: "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.580531 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.580497 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istiod-ca-cert" pod "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" (UID: "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:57.588557 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.588504 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-envoy" pod "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" (UID: "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.590758 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.590721 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-envoy" pod "3b8008c2-15fb-4bf0-80e2-575a4f60abe6" (UID: "3b8008c2-15fb-4bf0-80e2-575a4f60abe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.594417 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.594333 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-601aea-serving-cert\"" Oct 30 00:57:57.596427 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.596168 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-credential-socket" pod "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" (UID: "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.596553 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.596453 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-credential-socket" pod "6c47f328-0846-4b77-b15e-d4561701475f" (UID: "6c47f328-0846-4b77-b15e-d4561701475f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.597639 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.597214 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-data" pod "6c47f328-0846-4b77-b15e-d4561701475f" (UID: "6c47f328-0846-4b77-b15e-d4561701475f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.620100 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.615822 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-kube-api-access-rg8tc" pod "3b8008c2-15fb-4bf0-80e2-575a4f60abe6" (UID: "3b8008c2-15fb-4bf0-80e2-575a4f60abe6"). InnerVolumeSpecName "kube-api-access-rg8tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:57.620100 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.616055 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c47f328-0846-4b77-b15e-d4561701475f-istio-podinfo" pod "6c47f328-0846-4b77-b15e-d4561701475f" (UID: "6c47f328-0846-4b77-b15e-d4561701475f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:57:57.631222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.628986 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-podinfo" pod "3b8008c2-15fb-4bf0-80e2-575a4f60abe6" (UID: "3b8008c2-15fb-4bf0-80e2-575a4f60abe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:57:57.631222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.629088 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-token" pod "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" (UID: "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:57.631222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.629164 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-kube-api-access-rzkwv" pod "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" (UID: "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5"). InnerVolumeSpecName "kube-api-access-rzkwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:57.631222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.629219 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-podinfo" pod "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" (UID: "a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:57:57.631222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.629332 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-istio-token" pod "6c47f328-0846-4b77-b15e-d4561701475f" (UID: "6c47f328-0846-4b77-b15e-d4561701475f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:57.631222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.629641 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-kube-api-access-nnzx2" pod "6c47f328-0846-4b77-b15e-d4561701475f" (UID: "6c47f328-0846-4b77-b15e-d4561701475f"). InnerVolumeSpecName "kube-api-access-nnzx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:57.631690 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.631315 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-token" pod "3b8008c2-15fb-4bf0-80e2-575a4f60abe6" (UID: "3b8008c2-15fb-4bf0-80e2-575a4f60abe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:57.633794 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.632203 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-envoy" pod "6c47f328-0846-4b77-b15e-d4561701475f" (UID: "6c47f328-0846-4b77-b15e-d4561701475f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:57.660249 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.660208 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt"] Oct 30 00:57:57.660454 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.660296 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55"] Oct 30 00:57:57.672890 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.672843 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-maas-api-tls\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.673090 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.672968 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.673090 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673020 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbq67\" (UniqueName: \"kubernetes.io/projected/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-kube-api-access-bbq67\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.673344 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673319 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-metrics-certs\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.673561 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673540 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673588 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673606 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnzx2\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-kube-api-access-nnzx2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673619 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673634 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673648 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673660 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c47f328-0846-4b77-b15e-d4561701475f-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673675 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673687 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673700 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673713 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673727 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg8tc\" (UniqueName: \"kubernetes.io/projected/3b8008c2-15fb-4bf0-80e2-575a4f60abe6-kube-api-access-rg8tc\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673740 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzkwv\" (UniqueName: \"kubernetes.io/projected/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-kube-api-access-rzkwv\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673755 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673768 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673780 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c47f328-0846-4b77-b15e-d4561701475f-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673793 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.673959 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.673809 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c47f328-0846-4b77-b15e-d4561701475f-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:57:57.739830 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.739768 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55"] Oct 30 00:57:57.740002 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.739936 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.740082 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.740046 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8"] Oct 30 00:57:57.749387 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.749358 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8"] Oct 30 00:57:57.749632 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.749467 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.780865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.778469 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-metrics-certs\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.780865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.780545 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-maas-api-tls\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.780865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.780626 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.780865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.780683 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbq67\" (UniqueName: \"kubernetes.io/projected/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-kube-api-access-bbq67\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.783594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.783552 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.790450 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.790416 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-maas-api-tls\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.791184 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.791157 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-metrics-certs\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.794108 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.794065 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-73b32e-openshift-default-dockercfg-mknnz\"" Oct 30 00:57:57.802701 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:57:57.802663 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a9b83a_6f6f_44e3_81b8_6e363d9fcaa5.slice/crio-790314ade38637f40fc778b59b50a40fa47bb0cd9ab5fda1c9d69e410735c03d\": RecentStats: unable to find data in memory cache]" Oct 30 00:57:57.812116 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.812079 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbq67\" (UniqueName: \"kubernetes.io/projected/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-kube-api-access-bbq67\") pod \"maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:57.885562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.885522 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.885767 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.885608 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.885767 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.885642 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.885767 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.885761 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.885912 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.885808 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ef110e-b36f-42e1-a30c-399f16249151-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.885912 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.885843 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.886084 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886057 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.886148 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886133 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bmn\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-kube-api-access-p4bmn\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.886222 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886203 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.886331 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886303 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.886509 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886475 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.886630 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886581 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.886704 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886652 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ef110e-b36f-42e1-a30c-399f16249151-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.886769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886707 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.886769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886735 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.886769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886754 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.886889 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886846 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.886963 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.886932 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxl4\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-kube-api-access-xdxl4\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.988266 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988227 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bmn\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-kube-api-access-p4bmn\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.988467 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988279 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.988467 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988350 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.988467 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988405 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.988467 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988441 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.988711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988490 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ef110e-b36f-42e1-a30c-399f16249151-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.988711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988524 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.988711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988548 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.988711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988588 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.988711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988651 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.988711 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988700 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxl4\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-kube-api-access-xdxl4\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988716 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.989024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988793 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.989024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988799 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988848 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988885 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.989024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988939 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.988987 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.989040 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ef110e-b36f-42e1-a30c-399f16249151-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.989093 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.989215 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.989544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.989042 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.989276 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.989328 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.989855 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.989720 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.990060 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.989942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ef110e-b36f-42e1-a30c-399f16249151-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:57.990201 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.990178 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.996447 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.991153 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.996447 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.991852 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.996447 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.992082 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:57.996447 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:57.992646 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ef110e-b36f-42e1-a30c-399f16249151-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:58.010249 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.010215 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:58.012626 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.012536 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bmn\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-kube-api-access-p4bmn\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:58.012869 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.012805 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:58.012974 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.012904 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:58.013064 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.013041 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxl4\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-kube-api-access-xdxl4\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:58.052071 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.051991 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:58.081514 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.081475 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:57:58.093743 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.093468 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:57:58.340192 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.340102 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" Oct 30 00:57:58.340361 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.340102 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk" event={"ID":"6c47f328-0846-4b77-b15e-d4561701475f","Type":"ContainerDied","Data":"1a443c36a67259dbee57d5228366e5645dd38169bf94dfa57ab7d7fe9e30e80c"} Oct 30 00:57:58.340361 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.340280 4927 scope.go:122] "RemoveContainer" containerID="e7f04844e1d2d39ab1931c2efb4b792e1bd889f47f601721509818ae0135dc60" Oct 30 00:57:58.342042 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.342019 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" Oct 30 00:57:58.342143 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.342037 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb" event={"ID":"3b8008c2-15fb-4bf0-80e2-575a4f60abe6","Type":"ContainerDied","Data":"1ec3aa4ed3d88dde6892ca2d091b664e3be0922cc7a4d210c6ec6b99fc667d34"} Oct 30 00:57:58.373768 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.357758 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" event={"ID":"a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5","Type":"ContainerDied","Data":"790314ade38637f40fc778b59b50a40fa47bb0cd9ab5fda1c9d69e410735c03d"} Oct 30 00:57:58.373768 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.358049 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z" Oct 30 00:57:58.380183 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.380160 4927 scope.go:122] "RemoveContainer" containerID="9d784856571772591add28df8e0b90b090c921d7c694fb0895b1b5f430dacd5c" Oct 30 00:57:58.396164 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.396107 4927 scope.go:122] "RemoveContainer" containerID="da5e491af6571aa496e21a52b86556f7ccaa99788fe288a24a30d0757cf8a7ec" Oct 30 00:57:58.403529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.402355 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk"] Oct 30 00:57:58.406498 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:57:58.406467 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed5c5d1_269c_421a_b5b1_765afaf5b9aa.slice/crio-62d428bd610023a0e7d313c3d76f72a1598bfc5cafde7ade1ee1a8e0a974e953 WatchSource:0}: Error finding container 62d428bd610023a0e7d313c3d76f72a1598bfc5cafde7ade1ee1a8e0a974e953: Status 404 returned error can't find the container with id 62d428bd610023a0e7d313c3d76f72a1598bfc5cafde7ade1ee1a8e0a974e953 Oct 30 00:57:58.408910 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.408889 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:57:58.409550 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.409529 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt"] Oct 30 00:57:58.416296 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.416259 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-mj6fk"] Oct 30 00:57:58.448123 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.448091 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55"] Oct 30 00:57:58.448881 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:57:58.448846 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ef110e_b36f_42e1_a30c_399f16249151.slice/crio-27aee5166bfeb5ce8838bee399be90247101dcd59b0d8ee83dfacfa970caae86 WatchSource:0}: Error finding container 27aee5166bfeb5ce8838bee399be90247101dcd59b0d8ee83dfacfa970caae86: Status 404 returned error can't find the container with id 27aee5166bfeb5ce8838bee399be90247101dcd59b0d8ee83dfacfa970caae86 Oct 30 00:57:58.452756 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.452719 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:58.452868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.452811 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:58.452868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.452857 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:58.460624 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:57:58.460540 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb8a0de_60b0_42d8_bb7a_1e83e81db566.slice/crio-1c3db6c490a87200742f5a3be84dd6c1ee0c5d4ec5a19564d490fc7ecf2cf99c WatchSource:0}: Error finding container 1c3db6c490a87200742f5a3be84dd6c1ee0c5d4ec5a19564d490fc7ecf2cf99c: Status 404 returned error can't find the container with id 1c3db6c490a87200742f5a3be84dd6c1ee0c5d4ec5a19564d490fc7ecf2cf99c Oct 30 00:57:58.460960 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.460925 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8"] Oct 30 00:57:58.469218 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.469172 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:58.469373 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.469259 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:58.469373 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.469298 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:58.475668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.475639 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb"] Oct 30 00:57:58.481074 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.481045 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-s85xb"] Oct 30 00:57:58.514028 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.513958 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z"] Oct 30 00:57:58.521154 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:58.521119 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-vhm2z"] Oct 30 00:57:59.519744 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.519692 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" event={"ID":"81ef110e-b36f-42e1-a30c-399f16249151","Type":"ContainerStarted","Data":"30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48"} Oct 30 00:57:59.520392 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.520348 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" event={"ID":"81ef110e-b36f-42e1-a30c-399f16249151","Type":"ContainerStarted","Data":"27aee5166bfeb5ce8838bee399be90247101dcd59b0d8ee83dfacfa970caae86"} Oct 30 00:57:59.525339 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.525308 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" event={"ID":"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa","Type":"ContainerStarted","Data":"54631015de9dcc7f738a3735878a157cad15c16024d535b5fa365819165ceb0f"} Oct 30 00:57:59.525605 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.525561 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" event={"ID":"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa","Type":"ContainerStarted","Data":"62d428bd610023a0e7d313c3d76f72a1598bfc5cafde7ade1ee1a8e0a974e953"} Oct 30 00:57:59.526966 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.526946 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:57:59.538646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.538609 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" event={"ID":"bdb8a0de-60b0-42d8-bb7a-1e83e81db566","Type":"ContainerStarted","Data":"6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69"} Oct 30 00:57:59.538953 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.538930 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" event={"ID":"bdb8a0de-60b0-42d8-bb7a-1e83e81db566","Type":"ContainerStarted","Data":"1c3db6c490a87200742f5a3be84dd6c1ee0c5d4ec5a19564d490fc7ecf2cf99c"} Oct 30 00:57:59.640943 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.640874 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" podStartSLOduration=6.23770054 podStartE2EDuration="6.640854632s" totalImagesPullingTime="403.154092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:59.635774383 +0000 UTC m=+1418.207700606" watchObservedRunningTime="2026-10-30 00:57:59.640854632 +0000 UTC m=+1418.212780867" Oct 30 00:57:59.643226 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.643196 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8008c2-15fb-4bf0-80e2-575a4f60abe6" path="/var/lib/kubelet/pods/3b8008c2-15fb-4bf0-80e2-575a4f60abe6/volumes" Oct 30 00:57:59.643523 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.643488 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" podStartSLOduration=32.643474326 podStartE2EDuration="32.643474326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:59.585042887 +0000 UTC m=+1418.156969135" watchObservedRunningTime="2026-10-30 00:57:59.643474326 +0000 UTC m=+1418.215400548" Oct 30 00:57:59.645402 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.645377 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c47f328-0846-4b77-b15e-d4561701475f" path="/var/lib/kubelet/pods/6c47f328-0846-4b77-b15e-d4561701475f/volumes" Oct 30 00:57:59.646047 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.646026 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5" path="/var/lib/kubelet/pods/a1a9b83a-6f6f-44e3-81b8-6e363d9fcaa5/volumes" Oct 30 00:57:59.705598 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:57:59.698687 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" podStartSLOduration=32.698666189 podStartE2EDuration="32.698666189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:59.695373657 +0000 UTC m=+1418.267299862" watchObservedRunningTime="2026-10-30 00:57:59.698666189 +0000 UTC m=+1418.270592413" Oct 30 00:58:00.089595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:00.081979 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:58:00.089595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:00.088760 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:58:00.089595 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:00.088824 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 00:58:00.098596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:00.095311 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:58:00.098596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:00.095460 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:58:00.098596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:00.095504 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 00:58:01.086599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:01.083131 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:58:01.086599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:01.083199 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 00:58:01.095594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:01.094216 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:58:01.095594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:01.094286 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 00:58:02.114596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:02.091776 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:58:02.114596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:02.091852 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 00:58:02.114596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:02.105091 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:58:02.114596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:02.105156 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 00:58:02.532098 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:02.532055 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk"] Oct 30 00:58:02.532347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:02.532316 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" podUID="f2909829-1690-436a-819e-1c21db428990" containerName="maas-api" containerID="cri-o://84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb" gracePeriod=30 Oct 30 00:58:03.096596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:03.092993 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:58:03.096596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:03.093060 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 00:58:03.131695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:03.129742 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:58:03.131695 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:03.129807 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 00:58:03.978913 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:03.971407 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:58:04.086519 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.086483 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:58:04.113555 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.113502 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 00:58:04.121849 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.121798 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:58:04.122145 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.122101 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 00:58:04.166593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.148683 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-metrics-certs\") pod \"f2909829-1690-436a-819e-1c21db428990\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " Oct 30 00:58:04.167418 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.167388 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2909829-1690-436a-819e-1c21db428990-kube-api-access-9lggf\" (UniqueName: \"kubernetes.io/projected/f2909829-1690-436a-819e-1c21db428990-kube-api-access-9lggf\") pod \"f2909829-1690-436a-819e-1c21db428990\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " Oct 30 00:58:04.210597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.198551 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2909829-1690-436a-819e-1c21db428990-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2909829-1690-436a-819e-1c21db428990-openshift-service-ca\") pod \"f2909829-1690-436a-819e-1c21db428990\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " Oct 30 00:58:04.329258 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.329216 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-maas-api-tls\") pod \"f2909829-1690-436a-819e-1c21db428990\" (UID: \"f2909829-1690-436a-819e-1c21db428990\") " Oct 30 00:58:04.337123 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.236841 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2909829-1690-436a-819e-1c21db428990-openshift-service-ca" pod "f2909829-1690-436a-819e-1c21db428990" (UID: "f2909829-1690-436a-819e-1c21db428990"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:04.337123 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.293215 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2909829-1690-436a-819e-1c21db428990-kube-api-access-9lggf" pod "f2909829-1690-436a-819e-1c21db428990" (UID: "f2909829-1690-436a-819e-1c21db428990"). InnerVolumeSpecName "kube-api-access-9lggf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:04.358597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.346137 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2909829-1690-436a-819e-1c21db428990-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:04.358597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.358319 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lggf\" (UniqueName: \"kubernetes.io/projected/f2909829-1690-436a-819e-1c21db428990-kube-api-access-9lggf\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:04.358597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.350476 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" event={"ID":"f2909829-1690-436a-819e-1c21db428990","Type":"ContainerDied","Data":"84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb"} Oct 30 00:58:04.358597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.358403 4927 scope.go:122] "RemoveContainer" containerID="84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb" Oct 30 00:58:04.358597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.350443 4927 generic.go:356] "Generic (PLEG): container finished" podID="f2909829-1690-436a-819e-1c21db428990" containerID="84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb" exitCode=0 Oct 30 00:58:04.358957 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.358930 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" event={"ID":"f2909829-1690-436a-819e-1c21db428990","Type":"ContainerDied","Data":"099195db54a23b5f39c17ea3ef5c524f8551487eb15cf0057ddf2b2b3e4cb27a"} Oct 30 00:58:04.359092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.350550 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk" Oct 30 00:58:04.359998 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.291823 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:04.360658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.355395 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-maas-api-tls" pod "f2909829-1690-436a-819e-1c21db428990" (UID: "f2909829-1690-436a-819e-1c21db428990"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:04.360883 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.360862 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-metrics-certs" pod "f2909829-1690-436a-819e-1c21db428990" (UID: "f2909829-1690-436a-819e-1c21db428990"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:04.408980 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.404675 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq"] Oct 30 00:58:04.408980 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.406202 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f2909829-1690-436a-819e-1c21db428990" containerName="maas-api" Oct 30 00:58:04.408980 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.406228 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2909829-1690-436a-819e-1c21db428990" containerName="maas-api" Oct 30 00:58:04.408980 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.406523 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2909829-1690-436a-819e-1c21db428990" containerName="maas-api" Oct 30 00:58:04.441639 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.439975 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.446244 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.445973 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq"] Oct 30 00:58:04.449391 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.449365 4927 scope.go:122] "RemoveContainer" containerID="84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb" Oct 30 00:58:04.449676 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:58:04.449650 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb\": container with ID starting with 84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb not found: ID does not exist" containerID="84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb" Oct 30 00:58:04.449764 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.449686 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb"} err="failed to get container status \"84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb\": rpc error: code = NotFound desc = could not find container \"84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb\": container with ID starting with 84a92a0ca06b9ecc3ea86226012d1892b54115be2366252b159e2ab44e1dd9fb not found: ID does not exist" Oct 30 00:58:04.460730 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.460703 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:04.460730 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.460725 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2909829-1690-436a-819e-1c21db428990-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:04.461821 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.461798 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a32412-serving-cert\"" Oct 30 00:58:04.470208 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:58:04.470176 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:58:04.562722 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.562635 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e207b0f-0bc0-485c-9030-adc9b94a7eac-openshift-service-ca\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.563241 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.563088 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-metrics-certs\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.563621 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.563446 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drnmg\" (UniqueName: \"kubernetes.io/projected/4e207b0f-0bc0-485c-9030-adc9b94a7eac-kube-api-access-drnmg\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.563697 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.563665 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-maas-api-tls\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.665373 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.665324 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-metrics-certs\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.665557 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.665520 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnmg\" (UniqueName: \"kubernetes.io/projected/4e207b0f-0bc0-485c-9030-adc9b94a7eac-kube-api-access-drnmg\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.665623 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.665581 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-maas-api-tls\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.665688 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.665672 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e207b0f-0bc0-485c-9030-adc9b94a7eac-openshift-service-ca\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.666971 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.666942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e207b0f-0bc0-485c-9030-adc9b94a7eac-openshift-service-ca\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.669039 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.669014 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-maas-api-tls\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.669177 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.669123 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-metrics-certs\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.709484 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.709410 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnmg\" (UniqueName: \"kubernetes.io/projected/4e207b0f-0bc0-485c-9030-adc9b94a7eac-kube-api-access-drnmg\") pod \"maas-api-e2e-api-b-a32412-59fc67746b-bljhq\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.768342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.768194 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:04.862188 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.862155 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk"] Oct 30 00:58:04.868201 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:04.868167 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-jzgxk"] Oct 30 00:58:05.115865 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:05.093015 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:58:05.131125 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:05.117422 4927 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 00:58:05.131125 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:05.130687 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 00:58:05.146593 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:05.135660 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 00:58:05.250111 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:05.250069 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq"] Oct 30 00:58:05.271758 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:58:05.271719 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e207b0f_0bc0_485c_9030_adc9b94a7eac.slice/crio-e5a6fcadc76d61318485f83fc7c339e369fe7c66b265de660787cc17d1e73048 WatchSource:0}: Error finding container e5a6fcadc76d61318485f83fc7c339e369fe7c66b265de660787cc17d1e73048: Status 404 returned error can't find the container with id e5a6fcadc76d61318485f83fc7c339e369fe7c66b265de660787cc17d1e73048 Oct 30 00:58:05.370712 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:05.370604 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" event={"ID":"4e207b0f-0bc0-485c-9030-adc9b94a7eac","Type":"ContainerStarted","Data":"e5a6fcadc76d61318485f83fc7c339e369fe7c66b265de660787cc17d1e73048"} Oct 30 00:58:05.575534 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:05.575490 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2909829-1690-436a-819e-1c21db428990" path="/var/lib/kubelet/pods/f2909829-1690-436a-819e-1c21db428990/volumes" Oct 30 00:58:06.111284 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.111199 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:58:06.114351 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.114318 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:58:06.115999 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.115961 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 00:58:06.220372 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.220338 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:58:06.225668 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.221976 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:58:06.228249 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.228073 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 00:58:06.390287 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.390245 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" event={"ID":"4e207b0f-0bc0-485c-9030-adc9b94a7eac","Type":"ContainerStarted","Data":"11b49107e30410eb83c45c2ab42cd5ae35c71ef89cff02154f07da9b275fe5ec"} Oct 30 00:58:06.391098 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.391071 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:06.441045 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.440986 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" podStartSLOduration=37.043359823 podStartE2EDuration="37.440971035s" totalImagesPullingTime="397.611212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:06.435520785 +0000 UTC m=+1425.007447001" watchObservedRunningTime="2026-10-30 00:58:06.440971035 +0000 UTC m=+1425.012897276" Oct 30 00:58:06.648650 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:06.641495 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 00:58:13.412642 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:13.412607 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 00:58:34.446747 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:58:34.446707 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:58:34.998342 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:34.998296 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8"] Oct 30 00:58:34.998640 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:34.998606 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" containerID="cri-o://c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a" gracePeriod=30 Oct 30 00:58:35.009707 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:35.009667 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h"] Oct 30 00:58:35.009978 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:35.009946 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" containerID="cri-o://aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1" gracePeriod=30 Oct 30 00:58:35.305564 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:35.305457 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2"] Oct 30 00:58:35.305771 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:35.305740 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" containerID="cri-o://b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84" gracePeriod=30 Oct 30 00:58:35.311782 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:35.311738 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5"] Oct 30 00:58:35.312094 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:35.312060 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" containerID="cri-o://61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a" gracePeriod=30 Oct 30 00:58:35.317052 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:35.317024 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw"] Oct 30 00:58:35.317260 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:35.317238 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" containerID="cri-o://059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43" gracePeriod=30 Oct 30 00:58:40.521106 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.518995 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:58:40.522339 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.522310 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609193 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9960c4b-6496-450a-987f-95370d602e37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9960c4b-6496-450a-987f-95370d602e37-istiod-ca-cert\") pod \"d9960c4b-6496-450a-987f-95370d602e37\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609241 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-kube-api-access-5wxdr\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-kube-api-access-5wxdr\") pod \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609272 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-socket\") pod \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609320 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-data\") pod \"d9960c4b-6496-450a-987f-95370d602e37\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609351 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-certs\") pod \"d9960c4b-6496-450a-987f-95370d602e37\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609428 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-socket\") pod \"d9960c4b-6496-450a-987f-95370d602e37\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609469 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-envoy\") pod \"d9960c4b-6496-450a-987f-95370d602e37\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609489 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istiod-ca-cert\") pod \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609529 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-credential-socket\") pod \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609623 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-kube-api-access-cmcbx\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-kube-api-access-cmcbx\") pod \"d9960c4b-6496-450a-987f-95370d602e37\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609671 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-podinfo\") pod \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " Oct 30 00:58:40.609717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609720 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-token\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-token\") pod \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " Oct 30 00:58:40.610347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609819 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-certs\") pod \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " Oct 30 00:58:40.610347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609855 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-data\") pod \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " Oct 30 00:58:40.610347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609963 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9960c4b-6496-450a-987f-95370d602e37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9960c4b-6496-450a-987f-95370d602e37-istio-podinfo\") pod \"d9960c4b-6496-450a-987f-95370d602e37\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " Oct 30 00:58:40.610347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.609997 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-envoy\") pod \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\" (UID: \"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9\") " Oct 30 00:58:40.610347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.610033 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-istio-token\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-istio-token\") pod \"d9960c4b-6496-450a-987f-95370d602e37\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " Oct 30 00:58:40.610347 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.610056 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-credential-socket\") pod \"d9960c4b-6496-450a-987f-95370d602e37\" (UID: \"d9960c4b-6496-450a-987f-95370d602e37\") " Oct 30 00:58:40.611531 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.611498 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-credential-socket" pod "d9960c4b-6496-450a-987f-95370d602e37" (UID: "d9960c4b-6496-450a-987f-95370d602e37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.612022 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.611912 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9960c4b-6496-450a-987f-95370d602e37-istiod-ca-cert" pod "d9960c4b-6496-450a-987f-95370d602e37" (UID: "d9960c4b-6496-450a-987f-95370d602e37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:40.617974 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.615449 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-certs" pod "d9960c4b-6496-450a-987f-95370d602e37" (UID: "d9960c4b-6496-450a-987f-95370d602e37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.617974 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.615530 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-data" pod "d9960c4b-6496-450a-987f-95370d602e37" (UID: "d9960c4b-6496-450a-987f-95370d602e37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.617974 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.616044 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-kube-api-access-cmcbx" pod "d9960c4b-6496-450a-987f-95370d602e37" (UID: "d9960c4b-6496-450a-987f-95370d602e37"). InnerVolumeSpecName "kube-api-access-cmcbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.618245 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.617991 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-kube-api-access-5wxdr" pod "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" (UID: "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9"). InnerVolumeSpecName "kube-api-access-5wxdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.618913 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.618886 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-envoy" pod "d9960c4b-6496-450a-987f-95370d602e37" (UID: "d9960c4b-6496-450a-987f-95370d602e37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.619693 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.619668 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-socket" pod "d9960c4b-6496-450a-987f-95370d602e37" (UID: "d9960c4b-6496-450a-987f-95370d602e37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.620824 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.620782 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9960c4b-6496-450a-987f-95370d602e37-istio-podinfo" pod "d9960c4b-6496-450a-987f-95370d602e37" (UID: "d9960c4b-6496-450a-987f-95370d602e37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:58:40.621622 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.621588 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-istio-token" pod "d9960c4b-6496-450a-987f-95370d602e37" (UID: "d9960c4b-6496-450a-987f-95370d602e37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.622019 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.621993 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-socket" pod "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" (UID: "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.622111 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.622011 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-credential-socket" pod "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" (UID: "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.622701 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.622664 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-certs" pod "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" (UID: "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.622808 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.622698 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-data" pod "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" (UID: "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.623030 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.623009 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istiod-ca-cert" pod "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" (UID: "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:40.627765 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.627736 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-envoy" pod "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" (UID: "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.628442 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.628418 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-podinfo" pod "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" (UID: "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:58:40.628565 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.628472 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-token" pod "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" (UID: "9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.668544 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.668377 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713020 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-socket\") pod \"32ce3b57-9419-45e1-a7a2-548ea15e545a\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713097 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-kube-api-access-rbx5d\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-kube-api-access-rbx5d\") pod \"32ce3b57-9419-45e1-a7a2-548ea15e545a\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713202 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-credential-socket\") pod \"32ce3b57-9419-45e1-a7a2-548ea15e545a\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713242 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ce3b57-9419-45e1-a7a2-548ea15e545a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce3b57-9419-45e1-a7a2-548ea15e545a-istiod-ca-cert\") pod \"32ce3b57-9419-45e1-a7a2-548ea15e545a\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713372 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-certs\") pod \"32ce3b57-9419-45e1-a7a2-548ea15e545a\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713440 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-socket" pod "32ce3b57-9419-45e1-a7a2-548ea15e545a" (UID: "32ce3b57-9419-45e1-a7a2-548ea15e545a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713600 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-podinfo\") pod \"32ce3b57-9419-45e1-a7a2-548ea15e545a\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713638 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-token\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-token\") pod \"32ce3b57-9419-45e1-a7a2-548ea15e545a\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713681 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-credential-socket" pod "32ce3b57-9419-45e1-a7a2-548ea15e545a" (UID: "32ce3b57-9419-45e1-a7a2-548ea15e545a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713710 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-envoy\") pod \"32ce3b57-9419-45e1-a7a2-548ea15e545a\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " Oct 30 00:58:40.714315 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.713805 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-data\") pod \"32ce3b57-9419-45e1-a7a2-548ea15e545a\" (UID: \"32ce3b57-9419-45e1-a7a2-548ea15e545a\") " Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714742 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714761 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-certs" pod "32ce3b57-9419-45e1-a7a2-548ea15e545a" (UID: "32ce3b57-9419-45e1-a7a2-548ea15e545a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714766 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714806 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714824 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9960c4b-6496-450a-987f-95370d602e37-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714837 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714851 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714864 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714879 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9960c4b-6496-450a-987f-95370d602e37-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714944 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wxdr\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-kube-api-access-5wxdr\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714962 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714973 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714992 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.715000 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.715010 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9960c4b-6496-450a-987f-95370d602e37-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.714943 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-data" pod "32ce3b57-9419-45e1-a7a2-548ea15e545a" (UID: "32ce3b57-9419-45e1-a7a2-548ea15e545a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.715023 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.715037 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.715048 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmcbx\" (UniqueName: \"kubernetes.io/projected/d9960c4b-6496-450a-987f-95370d602e37-kube-api-access-cmcbx\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.715058 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.715067 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.715058 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.715076 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.718397 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.715712 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ce3b57-9419-45e1-a7a2-548ea15e545a-istiod-ca-cert" pod "32ce3b57-9419-45e1-a7a2-548ea15e545a" (UID: "32ce3b57-9419-45e1-a7a2-548ea15e545a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:40.721024 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.720971 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-kube-api-access-rbx5d" pod "32ce3b57-9419-45e1-a7a2-548ea15e545a" (UID: "32ce3b57-9419-45e1-a7a2-548ea15e545a"). InnerVolumeSpecName "kube-api-access-rbx5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.725821 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.725771 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-token" pod "32ce3b57-9419-45e1-a7a2-548ea15e545a" (UID: "32ce3b57-9419-45e1-a7a2-548ea15e545a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.729646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.729555 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-envoy" pod "32ce3b57-9419-45e1-a7a2-548ea15e545a" (UID: "32ce3b57-9419-45e1-a7a2-548ea15e545a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.737886 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.736369 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-podinfo" pod "32ce3b57-9419-45e1-a7a2-548ea15e545a" (UID: "32ce3b57-9419-45e1-a7a2-548ea15e545a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:58:40.786791 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.786759 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:58:40.791245 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.791218 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:58:40.810352 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.810309 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44"] Oct 30 00:58:40.811354 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811326 4927 cpu_manager.go:414] "containerMap: removing container" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" Oct 30 00:58:40.811432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811355 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" Oct 30 00:58:40.811432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811375 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" Oct 30 00:58:40.811432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811381 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" Oct 30 00:58:40.811432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811392 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" Oct 30 00:58:40.811432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811397 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" Oct 30 00:58:40.811432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811419 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" Oct 30 00:58:40.811432 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811425 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" Oct 30 00:58:40.811672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811439 4927 cpu_manager.go:414] "containerMap: removing container" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" Oct 30 00:58:40.811672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811444 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" Oct 30 00:58:40.811672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811600 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerName="istio-proxy" Oct 30 00:58:40.811672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811616 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9960c4b-6496-450a-987f-95370d602e37" containerName="istio-proxy" Oct 30 00:58:40.811672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811626 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerName="istio-proxy" Oct 30 00:58:40.811672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811634 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerName="istio-proxy" Oct 30 00:58:40.811672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.811642 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" containerName="istio-proxy" Oct 30 00:58:40.815868 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.815843 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:40.816826 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.816796 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce3b57-9419-45e1-a7a2-548ea15e545a-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.816826 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.816824 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.817012 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.816833 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.817012 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.816841 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.817012 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.816849 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.817012 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.816857 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce3b57-9419-45e1-a7a2-548ea15e545a-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.817012 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.816865 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbx5d\" (UniqueName: \"kubernetes.io/projected/32ce3b57-9419-45e1-a7a2-548ea15e545a-kube-api-access-rbx5d\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.822354 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.822321 4927 generic.go:356] "Generic (PLEG): container finished" podID="d9960c4b-6496-450a-987f-95370d602e37" containerID="c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a" exitCode=0 Oct 30 00:58:40.822493 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.822413 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" Oct 30 00:58:40.822493 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.822441 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" event={"ID":"d9960c4b-6496-450a-987f-95370d602e37","Type":"ContainerDied","Data":"c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a"} Oct 30 00:58:40.822721 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.822496 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8" event={"ID":"d9960c4b-6496-450a-987f-95370d602e37","Type":"ContainerDied","Data":"684bed5899ce2a5ff0cbf07cfc8cce328360472e63ad8a6f8ce24bbb8ee120f2"} Oct 30 00:58:40.822721 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.822527 4927 scope.go:122] "RemoveContainer" containerID="c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a" Oct 30 00:58:40.824497 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.824472 4927 generic.go:356] "Generic (PLEG): container finished" podID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" containerID="aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1" exitCode=0 Oct 30 00:58:40.824701 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.824553 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" Oct 30 00:58:40.824795 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.824564 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" event={"ID":"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9","Type":"ContainerDied","Data":"aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1"} Oct 30 00:58:40.824903 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.824849 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h" event={"ID":"9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9","Type":"ContainerDied","Data":"b9a915feffb1338b9c005eb171171abd3d03ea6e03111fbb846b7fd08999c7bb"} Oct 30 00:58:40.843277 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.834361 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d8090513-serving-cert\"" Oct 30 00:58:40.860845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.856101 4927 generic.go:356] "Generic (PLEG): container finished" podID="de24b267-3437-4138-90fa-fbf49824d0a4" containerID="b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84" exitCode=0 Oct 30 00:58:40.860845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.856293 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" event={"ID":"de24b267-3437-4138-90fa-fbf49824d0a4","Type":"ContainerDied","Data":"b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84"} Oct 30 00:58:40.860845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.856363 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" event={"ID":"de24b267-3437-4138-90fa-fbf49824d0a4","Type":"ContainerDied","Data":"0758d428fcbf6f7fb39632b6e99df34a3158b18dc134bb79d091dd4fb44fe33e"} Oct 30 00:58:40.860845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.856460 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2" Oct 30 00:58:40.860845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.860288 4927 generic.go:356] "Generic (PLEG): container finished" podID="c0686fba-808a-40ab-9317-ed3664c7b7f4" containerID="61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a" exitCode=0 Oct 30 00:58:40.860845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.860417 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" event={"ID":"c0686fba-808a-40ab-9317-ed3664c7b7f4","Type":"ContainerDied","Data":"61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a"} Oct 30 00:58:40.860845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.860504 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" event={"ID":"c0686fba-808a-40ab-9317-ed3664c7b7f4","Type":"ContainerDied","Data":"9d18b67f55554f19291aa715b76c4338623adadfce9899009a14648ea84f2920"} Oct 30 00:58:40.860845 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.860616 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5" Oct 30 00:58:40.878627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.873895 4927 generic.go:356] "Generic (PLEG): container finished" podID="32ce3b57-9419-45e1-a7a2-548ea15e545a" containerID="059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43" exitCode=0 Oct 30 00:58:40.878627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.873975 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" event={"ID":"32ce3b57-9419-45e1-a7a2-548ea15e545a","Type":"ContainerDied","Data":"059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43"} Oct 30 00:58:40.878627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.874016 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" event={"ID":"32ce3b57-9419-45e1-a7a2-548ea15e545a","Type":"ContainerDied","Data":"e7368fc9bf958d3578e03904f1f294daf36df09c7dbe838f555af5461f797243"} Oct 30 00:58:40.878627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.874081 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw" Oct 30 00:58:40.878627 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.876755 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44"] Oct 30 00:58:40.881471 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.881445 4927 scope.go:122] "RemoveContainer" containerID="c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a" Oct 30 00:58:40.882000 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:58:40.881965 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a\": container with ID starting with c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a not found: ID does not exist" containerID="c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a" Oct 30 00:58:40.882136 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.882013 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a"} err="failed to get container status \"c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a\": rpc error: code = NotFound desc = could not find container \"c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a\": container with ID starting with c796193f98e55abbd1edaf69813820d16d90b5502ae6e8378c4030892c6bf87a not found: ID does not exist" Oct 30 00:58:40.882136 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.882040 4927 scope.go:122] "RemoveContainer" containerID="aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1" Oct 30 00:58:40.902231 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.901932 4927 scope.go:122] "RemoveContainer" containerID="aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1" Oct 30 00:58:40.902643 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:58:40.902344 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1\": container with ID starting with aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1 not found: ID does not exist" containerID="aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1" Oct 30 00:58:40.902643 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.902377 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1"} err="failed to get container status \"aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1\": rpc error: code = NotFound desc = could not find container \"aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1\": container with ID starting with aafa98f918b1b6f1f4fde9a57b24434a5e885b20bbed39bb6607fd8b131649c1 not found: ID does not exist" Oct 30 00:58:40.902643 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.902409 4927 scope.go:122] "RemoveContainer" containerID="b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84" Oct 30 00:58:40.918184 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918150 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-data\") pod \"c0686fba-808a-40ab-9317-ed3664c7b7f4\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " Oct 30 00:58:40.918360 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918208 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de24b267-3437-4138-90fa-fbf49824d0a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de24b267-3437-4138-90fa-fbf49824d0a4-istio-podinfo\") pod \"de24b267-3437-4138-90fa-fbf49824d0a4\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " Oct 30 00:58:40.918360 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918329 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-socket\") pod \"c0686fba-808a-40ab-9317-ed3664c7b7f4\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " Oct 30 00:58:40.918484 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918389 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-kube-api-access-hfhmc\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-kube-api-access-hfhmc\") pod \"c0686fba-808a-40ab-9317-ed3664c7b7f4\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " Oct 30 00:58:40.918484 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918436 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-data\") pod \"de24b267-3437-4138-90fa-fbf49824d0a4\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " Oct 30 00:58:40.918484 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918472 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-podinfo\") pod \"c0686fba-808a-40ab-9317-ed3664c7b7f4\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " Oct 30 00:58:40.918718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918553 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-envoy\") pod \"de24b267-3437-4138-90fa-fbf49824d0a4\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " Oct 30 00:58:40.918718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918603 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-certs\") pod \"de24b267-3437-4138-90fa-fbf49824d0a4\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " Oct 30 00:58:40.918718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918648 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-certs\") pod \"c0686fba-808a-40ab-9317-ed3664c7b7f4\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " Oct 30 00:58:40.918718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918689 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-credential-socket\") pod \"de24b267-3437-4138-90fa-fbf49824d0a4\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " Oct 30 00:58:40.918913 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918765 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-socket\") pod \"de24b267-3437-4138-90fa-fbf49824d0a4\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " Oct 30 00:58:40.918913 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918861 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-envoy\") pod \"c0686fba-808a-40ab-9317-ed3664c7b7f4\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " Oct 30 00:58:40.919013 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918956 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-token\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-token\") pod \"c0686fba-808a-40ab-9317-ed3664c7b7f4\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " Oct 30 00:58:40.919013 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.918991 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0686fba-808a-40ab-9317-ed3664c7b7f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0686fba-808a-40ab-9317-ed3664c7b7f4-istiod-ca-cert\") pod \"c0686fba-808a-40ab-9317-ed3664c7b7f4\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " Oct 30 00:58:40.919107 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.919030 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de24b267-3437-4138-90fa-fbf49824d0a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de24b267-3437-4138-90fa-fbf49824d0a4-istiod-ca-cert\") pod \"de24b267-3437-4138-90fa-fbf49824d0a4\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " Oct 30 00:58:40.919205 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.919150 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-istio-token\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-istio-token\") pod \"de24b267-3437-4138-90fa-fbf49824d0a4\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " Oct 30 00:58:40.919267 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.919218 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-credential-socket\") pod \"c0686fba-808a-40ab-9317-ed3664c7b7f4\" (UID: \"c0686fba-808a-40ab-9317-ed3664c7b7f4\") " Oct 30 00:58:40.919267 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.919255 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-kube-api-access-78r67\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-kube-api-access-78r67\") pod \"de24b267-3437-4138-90fa-fbf49824d0a4\" (UID: \"de24b267-3437-4138-90fa-fbf49824d0a4\") " Oct 30 00:58:40.919852 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.919666 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-maas-api-tls\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:40.919852 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.919674 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-certs" pod "c0686fba-808a-40ab-9317-ed3664c7b7f4" (UID: "c0686fba-808a-40ab-9317-ed3664c7b7f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.919852 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.919751 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-metrics-certs\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:40.920081 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.919860 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/685d1eb5-c754-4840-93af-583859a64759-openshift-service-ca\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:40.920081 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.919993 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v745f\" (UniqueName: \"kubernetes.io/projected/685d1eb5-c754-4840-93af-583859a64759-kube-api-access-v745f\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:40.920192 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.920138 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:40.920408 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.920298 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-data" pod "c0686fba-808a-40ab-9317-ed3664c7b7f4" (UID: "c0686fba-808a-40ab-9317-ed3664c7b7f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.920408 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.920361 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-credential-socket" pod "de24b267-3437-4138-90fa-fbf49824d0a4" (UID: "de24b267-3437-4138-90fa-fbf49824d0a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.920903 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.920729 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-socket" pod "de24b267-3437-4138-90fa-fbf49824d0a4" (UID: "de24b267-3437-4138-90fa-fbf49824d0a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.921250 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.921074 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-data" pod "de24b267-3437-4138-90fa-fbf49824d0a4" (UID: "de24b267-3437-4138-90fa-fbf49824d0a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.921337 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.921274 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-socket" pod "c0686fba-808a-40ab-9317-ed3664c7b7f4" (UID: "c0686fba-808a-40ab-9317-ed3664c7b7f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.921611 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.921420 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0686fba-808a-40ab-9317-ed3664c7b7f4-istiod-ca-cert" pod "c0686fba-808a-40ab-9317-ed3664c7b7f4" (UID: "c0686fba-808a-40ab-9317-ed3664c7b7f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:40.921699 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.921681 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-credential-socket" pod "c0686fba-808a-40ab-9317-ed3664c7b7f4" (UID: "c0686fba-808a-40ab-9317-ed3664c7b7f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.923769 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.923553 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de24b267-3437-4138-90fa-fbf49824d0a4-istiod-ca-cert" pod "de24b267-3437-4138-90fa-fbf49824d0a4" (UID: "de24b267-3437-4138-90fa-fbf49824d0a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:40.923909 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.923886 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-certs" pod "de24b267-3437-4138-90fa-fbf49824d0a4" (UID: "de24b267-3437-4138-90fa-fbf49824d0a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.929671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.928488 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de24b267-3437-4138-90fa-fbf49824d0a4-istio-podinfo" pod "de24b267-3437-4138-90fa-fbf49824d0a4" (UID: "de24b267-3437-4138-90fa-fbf49824d0a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:58:40.929671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.928658 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-istio-token" pod "de24b267-3437-4138-90fa-fbf49824d0a4" (UID: "de24b267-3437-4138-90fa-fbf49824d0a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.929671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.928747 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-kube-api-access-78r67" pod "de24b267-3437-4138-90fa-fbf49824d0a4" (UID: "de24b267-3437-4138-90fa-fbf49824d0a4"). InnerVolumeSpecName "kube-api-access-78r67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.929671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.928864 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-token" pod "c0686fba-808a-40ab-9317-ed3664c7b7f4" (UID: "c0686fba-808a-40ab-9317-ed3664c7b7f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.929671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.928950 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-envoy" pod "c0686fba-808a-40ab-9317-ed3664c7b7f4" (UID: "c0686fba-808a-40ab-9317-ed3664c7b7f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.929671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.929152 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-envoy" pod "de24b267-3437-4138-90fa-fbf49824d0a4" (UID: "de24b267-3437-4138-90fa-fbf49824d0a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:58:40.930436 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.930399 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-podinfo" pod "c0686fba-808a-40ab-9317-ed3664c7b7f4" (UID: "c0686fba-808a-40ab-9317-ed3664c7b7f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:58:40.930937 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.930904 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-kube-api-access-hfhmc" pod "c0686fba-808a-40ab-9317-ed3664c7b7f4" (UID: "c0686fba-808a-40ab-9317-ed3664c7b7f4"). InnerVolumeSpecName "kube-api-access-hfhmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:40.933803 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.933782 4927 scope.go:122] "RemoveContainer" containerID="b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84" Oct 30 00:58:40.934257 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:58:40.934226 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84\": container with ID starting with b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84 not found: ID does not exist" containerID="b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84" Oct 30 00:58:40.934298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.934259 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84"} err="failed to get container status \"b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84\": rpc error: code = NotFound desc = could not find container \"b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84\": container with ID starting with b1783dc206077302b01cf108c4f334c877d428bdeeb0435346fb8b848cabea84 not found: ID does not exist" Oct 30 00:58:40.934298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.934279 4927 scope.go:122] "RemoveContainer" containerID="61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a" Oct 30 00:58:40.948645 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.948614 4927 scope.go:122] "RemoveContainer" containerID="61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a" Oct 30 00:58:40.949073 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:58:40.949050 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a\": container with ID starting with 61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a not found: ID does not exist" containerID="61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a" Oct 30 00:58:40.949183 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.949079 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a"} err="failed to get container status \"61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a\": rpc error: code = NotFound desc = could not find container \"61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a\": container with ID starting with 61d691bf13bd91a93cd936414b2877244c845697cef0408fcda44614e220fa4a not found: ID does not exist" Oct 30 00:58:40.949183 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.949104 4927 scope.go:122] "RemoveContainer" containerID="059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43" Oct 30 00:58:40.962145 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.962121 4927 scope.go:122] "RemoveContainer" containerID="059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43" Oct 30 00:58:40.962675 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:58:40.962644 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43\": container with ID starting with 059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43 not found: ID does not exist" containerID="059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43" Oct 30 00:58:40.962805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:40.962682 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43"} err="failed to get container status \"059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43\": rpc error: code = NotFound desc = could not find container \"059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43\": container with ID starting with 059dc668f6c56894fcc0281188e5915725b57d2ce68e1149f8c5b2adb1c83d43 not found: ID does not exist" Oct 30 00:58:41.024600 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.024547 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/685d1eb5-c754-4840-93af-583859a64759-openshift-service-ca\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:41.025365 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.025180 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw"] Oct 30 00:58:41.026718 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.026694 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v745f\" (UniqueName: \"kubernetes.io/projected/685d1eb5-c754-4840-93af-583859a64759-kube-api-access-v745f\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:41.027739 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.027716 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/685d1eb5-c754-4840-93af-583859a64759-openshift-service-ca\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:41.028478 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.028013 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-maas-api-tls\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:41.028717 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.028698 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-metrics-certs\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:41.029480 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.029460 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.029614 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.029599 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.029731 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.029715 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.029816 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.029806 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0686fba-808a-40ab-9317-ed3664c7b7f4-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.029912 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.029899 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de24b267-3437-4138-90fa-fbf49824d0a4-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030002 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.029991 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030087 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030076 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030173 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030161 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78r67\" (UniqueName: \"kubernetes.io/projected/de24b267-3437-4138-90fa-fbf49824d0a4-kube-api-access-78r67\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030264 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030252 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030346 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030335 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de24b267-3437-4138-90fa-fbf49824d0a4-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030519 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030505 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0686fba-808a-40ab-9317-ed3664c7b7f4-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030637 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030622 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhmc\" (UniqueName: \"kubernetes.io/projected/c0686fba-808a-40ab-9317-ed3664c7b7f4-kube-api-access-hfhmc\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030730 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030719 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030811 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030802 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0686fba-808a-40ab-9317-ed3664c7b7f4-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030889 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030879 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.030972 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.030960 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.031613 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.031041 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de24b267-3437-4138-90fa-fbf49824d0a4-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:58:41.034049 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.034025 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-metrics-certs\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:41.034929 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.034905 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-gqfmw"] Oct 30 00:58:41.036272 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.036244 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-maas-api-tls\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:41.049797 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.049766 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v745f\" (UniqueName: \"kubernetes.io/projected/685d1eb5-c754-4840-93af-583859a64759-kube-api-access-v745f\") pod \"maas-api-e2e-mt-d8090513-784b489945-6tf44\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:41.115354 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.113924 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8"] Oct 30 00:58:41.115646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.115620 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kn9f8"] Oct 30 00:58:41.151502 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.151420 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h"] Oct 30 00:58:41.152243 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.152028 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:41.228298 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.226405 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-kpx7h"] Oct 30 00:58:41.352326 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.352293 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5"] Oct 30 00:58:41.357667 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.357636 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-kmrg5"] Oct 30 00:58:41.392636 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.391620 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2"] Oct 30 00:58:41.398292 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.398243 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-cgps2"] Oct 30 00:58:41.575486 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.575454 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ce3b57-9419-45e1-a7a2-548ea15e545a" path="/var/lib/kubelet/pods/32ce3b57-9419-45e1-a7a2-548ea15e545a/volumes" Oct 30 00:58:41.576073 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.576055 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9" path="/var/lib/kubelet/pods/9ebbe623-d4ec-48c0-bd54-ad8f53f3bac9/volumes" Oct 30 00:58:41.576585 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.576558 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0686fba-808a-40ab-9317-ed3664c7b7f4" path="/var/lib/kubelet/pods/c0686fba-808a-40ab-9317-ed3664c7b7f4/volumes" Oct 30 00:58:41.577406 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.577388 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9960c4b-6496-450a-987f-95370d602e37" path="/var/lib/kubelet/pods/d9960c4b-6496-450a-987f-95370d602e37/volumes" Oct 30 00:58:41.577910 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.577895 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de24b267-3437-4138-90fa-fbf49824d0a4" path="/var/lib/kubelet/pods/de24b267-3437-4138-90fa-fbf49824d0a4/volumes" Oct 30 00:58:41.614192 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.614154 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44"] Oct 30 00:58:41.625478 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:58:41.625429 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685d1eb5_c754_4840_93af_583859a64759.slice/crio-354a0f8b19c4d9349fa34da180185135d0bdd919af3235a2b1c935c349d4470c WatchSource:0}: Error finding container 354a0f8b19c4d9349fa34da180185135d0bdd919af3235a2b1c935c349d4470c: Status 404 returned error can't find the container with id 354a0f8b19c4d9349fa34da180185135d0bdd919af3235a2b1c935c349d4470c Oct 30 00:58:41.884468 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:41.884427 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" event={"ID":"685d1eb5-c754-4840-93af-583859a64759","Type":"ContainerStarted","Data":"354a0f8b19c4d9349fa34da180185135d0bdd919af3235a2b1c935c349d4470c"} Oct 30 00:58:42.950828 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:42.949373 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" event={"ID":"685d1eb5-c754-4840-93af-583859a64759","Type":"ContainerStarted","Data":"1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0"} Oct 30 00:58:42.950828 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:42.950763 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:49.974216 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:49.974183 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:58:50.020925 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:58:50.020855 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" podStartSLOduration=9.617640955 podStartE2EDuration="10.02083913s" totalImagesPullingTime="403.198175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:43.081670029 +0000 UTC m=+1461.653596251" watchObservedRunningTime="2026-10-30 00:58:50.02083913 +0000 UTC m=+1468.592765449" Oct 30 00:59:04.446255 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:59:04.446169 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:59:08.377829 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:08.377792 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6"] Oct 30 00:59:08.409851 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:08.409815 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh"] Oct 30 00:59:13.378652 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.378610 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" podUID="b1b442ce-b4dd-4d11-bfce-de988c90e482" containerName="payload-processing" containerID="cri-o://5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625" gracePeriod=25 Oct 30 00:59:13.410635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.410599 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" podUID="8616c4e2-8249-435d-a67e-4703af0cadf1" containerName="payload-pre-processing" containerID="cri-o://a8805b77d35f91be3d476c62585f80389001d6e5e65300397bd0fbbac8b7de0e" gracePeriod=25 Oct 30 00:59:13.785821 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.785794 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:59:13.809308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.808951 4927 generic.go:356] "Generic (PLEG): container finished" podID="b1b442ce-b4dd-4d11-bfce-de988c90e482" containerID="5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625" exitCode=0 Oct 30 00:59:13.809308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.808990 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" event={"ID":"b1b442ce-b4dd-4d11-bfce-de988c90e482","Type":"ContainerDied","Data":"5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625"} Oct 30 00:59:13.809308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.809025 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" Oct 30 00:59:13.809308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.809059 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6" event={"ID":"b1b442ce-b4dd-4d11-bfce-de988c90e482","Type":"ContainerDied","Data":"1fef3492cd6965200d33bd770bcddb3ea4d2dc23f9288c345e0b57017632b266"} Oct 30 00:59:13.809308 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.809100 4927 scope.go:122] "RemoveContainer" containerID="5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625" Oct 30 00:59:13.811355 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.811324 4927 generic.go:356] "Generic (PLEG): container finished" podID="8616c4e2-8249-435d-a67e-4703af0cadf1" containerID="a8805b77d35f91be3d476c62585f80389001d6e5e65300397bd0fbbac8b7de0e" exitCode=0 Oct 30 00:59:13.811480 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.811382 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" event={"ID":"8616c4e2-8249-435d-a67e-4703af0cadf1","Type":"ContainerDied","Data":"a8805b77d35f91be3d476c62585f80389001d6e5e65300397bd0fbbac8b7de0e"} Oct 30 00:59:13.849734 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.849313 4927 scope.go:122] "RemoveContainer" containerID="5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625" Oct 30 00:59:13.851403 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:59:13.851378 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625\": container with ID starting with 5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625 not found: ID does not exist" containerID="5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625" Oct 30 00:59:13.851547 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.851410 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625"} err="failed to get container status \"5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625\": rpc error: code = NotFound desc = could not find container \"5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625\": container with ID starting with 5a053c505bd6a0ba219d763c8688f86af8e2df00c949131406bb5d6281c22625 not found: ID does not exist" Oct 30 00:59:13.856126 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.856098 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b442ce-b4dd-4d11-bfce-de988c90e482-kube-api-access-25r4g\" (UniqueName: \"kubernetes.io/projected/b1b442ce-b4dd-4d11-bfce-de988c90e482-kube-api-access-25r4g\") pod \"b1b442ce-b4dd-4d11-bfce-de988c90e482\" (UID: \"b1b442ce-b4dd-4d11-bfce-de988c90e482\") " Oct 30 00:59:13.856285 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.856138 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:59:13.856285 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.856264 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b442ce-b4dd-4d11-bfce-de988c90e482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b442ce-b4dd-4d11-bfce-de988c90e482-plugins-config-volume\") pod \"b1b442ce-b4dd-4d11-bfce-de988c90e482\" (UID: \"b1b442ce-b4dd-4d11-bfce-de988c90e482\") " Oct 30 00:59:13.856652 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.856631 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b442ce-b4dd-4d11-bfce-de988c90e482-plugins-config-volume" pod "b1b442ce-b4dd-4d11-bfce-de988c90e482" (UID: "b1b442ce-b4dd-4d11-bfce-de988c90e482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:13.857360 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.857336 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b442ce-b4dd-4d11-bfce-de988c90e482-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:59:13.858996 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.858970 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b442ce-b4dd-4d11-bfce-de988c90e482-kube-api-access-25r4g" pod "b1b442ce-b4dd-4d11-bfce-de988c90e482" (UID: "b1b442ce-b4dd-4d11-bfce-de988c90e482"). InnerVolumeSpecName "kube-api-access-25r4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:13.958316 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.958219 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8616c4e2-8249-435d-a67e-4703af0cadf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8616c4e2-8249-435d-a67e-4703af0cadf1-plugins-config-volume\") pod \"8616c4e2-8249-435d-a67e-4703af0cadf1\" (UID: \"8616c4e2-8249-435d-a67e-4703af0cadf1\") " Oct 30 00:59:13.958617 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.958556 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8616c4e2-8249-435d-a67e-4703af0cadf1-kube-api-access-r5lkr\" (UniqueName: \"kubernetes.io/projected/8616c4e2-8249-435d-a67e-4703af0cadf1-kube-api-access-r5lkr\") pod \"8616c4e2-8249-435d-a67e-4703af0cadf1\" (UID: \"8616c4e2-8249-435d-a67e-4703af0cadf1\") " Oct 30 00:59:13.958801 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.958700 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8616c4e2-8249-435d-a67e-4703af0cadf1-plugins-config-volume" pod "8616c4e2-8249-435d-a67e-4703af0cadf1" (UID: "8616c4e2-8249-435d-a67e-4703af0cadf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:13.959300 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.959276 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25r4g\" (UniqueName: \"kubernetes.io/projected/b1b442ce-b4dd-4d11-bfce-de988c90e482-kube-api-access-25r4g\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:59:13.959300 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.959297 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8616c4e2-8249-435d-a67e-4703af0cadf1-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:59:13.961093 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:13.961065 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8616c4e2-8249-435d-a67e-4703af0cadf1-kube-api-access-r5lkr" pod "8616c4e2-8249-435d-a67e-4703af0cadf1" (UID: "8616c4e2-8249-435d-a67e-4703af0cadf1"). InnerVolumeSpecName "kube-api-access-r5lkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:14.060796 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:14.060767 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5lkr\" (UniqueName: \"kubernetes.io/projected/8616c4e2-8249-435d-a67e-4703af0cadf1-kube-api-access-r5lkr\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:59:14.155953 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:14.155911 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6"] Oct 30 00:59:14.161448 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:14.161419 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a4fe0143-6b46b4f5b4-b22r6"] Oct 30 00:59:14.821001 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:14.820915 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" Oct 30 00:59:14.821469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:14.820919 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh" event={"ID":"8616c4e2-8249-435d-a67e-4703af0cadf1","Type":"ContainerDied","Data":"b66d4b957982a636e1ec3612d6b92067e3572d711aa3ae52b1bc52ec4e55111d"} Oct 30 00:59:14.821469 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:14.821070 4927 scope.go:122] "RemoveContainer" containerID="a8805b77d35f91be3d476c62585f80389001d6e5e65300397bd0fbbac8b7de0e" Oct 30 00:59:14.873104 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:14.873074 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh"] Oct 30 00:59:14.884152 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:14.884124 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a4fe0143-d94c99447-4lmxh"] Oct 30 00:59:15.574000 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:15.573964 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8616c4e2-8249-435d-a67e-4703af0cadf1" path="/var/lib/kubelet/pods/8616c4e2-8249-435d-a67e-4703af0cadf1/volumes" Oct 30 00:59:15.574612 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:15.574583 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b442ce-b4dd-4d11-bfce-de988c90e482" path="/var/lib/kubelet/pods/b1b442ce-b4dd-4d11-bfce-de988c90e482/volumes" Oct 30 00:59:17.557822 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.557782 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv"] Oct 30 00:59:17.558946 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.558922 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8616c4e2-8249-435d-a67e-4703af0cadf1" containerName="payload-pre-processing" Oct 30 00:59:17.559110 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.559090 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8616c4e2-8249-435d-a67e-4703af0cadf1" containerName="payload-pre-processing" Oct 30 00:59:17.559233 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.559127 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b1b442ce-b4dd-4d11-bfce-de988c90e482" containerName="payload-processing" Oct 30 00:59:17.559233 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.559134 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b442ce-b4dd-4d11-bfce-de988c90e482" containerName="payload-processing" Oct 30 00:59:17.559358 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.559342 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b442ce-b4dd-4d11-bfce-de988c90e482" containerName="payload-processing" Oct 30 00:59:17.559415 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.559382 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8616c4e2-8249-435d-a67e-4703af0cadf1" containerName="payload-pre-processing" Oct 30 00:59:17.572341 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.572308 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.600562 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.600531 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv"] Oct 30 00:59:17.603145 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.603117 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5f4f9b-openshift-default-dockercfg-w9b79\"" Oct 30 00:59:17.653774 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.653739 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-certs\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.653774 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.653779 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-istio-token\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.654048 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.653826 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-envoy\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.654048 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.653994 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-data\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.654048 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.654047 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nx7\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-kube-api-access-z4nx7\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.654213 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.654111 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-credential-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.654267 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.654218 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34591e8b-c9d0-4e40-892b-732be072817e-istio-podinfo\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.654320 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.654272 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.654372 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.654350 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34591e8b-c9d0-4e40-892b-732be072817e-istiod-ca-cert\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.761596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.760850 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-data\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.761596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.760965 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nx7\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-kube-api-access-z4nx7\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.761596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.761038 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-credential-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.761596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.761133 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34591e8b-c9d0-4e40-892b-732be072817e-istio-podinfo\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.761596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.761202 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.761596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.761284 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34591e8b-c9d0-4e40-892b-732be072817e-istiod-ca-cert\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.761596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.761380 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-certs\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.761596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.761409 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-istio-token\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.761596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.761475 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-envoy\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.769604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.763839 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-data\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.769604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.764785 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-credential-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.769604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.768102 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.769604 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.769126 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34591e8b-c9d0-4e40-892b-732be072817e-istiod-ca-cert\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.778665 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.770010 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-certs\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.817646 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.810160 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-istio-token\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.830597 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.829168 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34591e8b-c9d0-4e40-892b-732be072817e-istio-podinfo\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.855599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.842613 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nx7\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-kube-api-access-z4nx7\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.855599 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.846681 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-envoy\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:17.907440 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:17.907404 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:18.476863 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:59:18.476827 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34591e8b_c9d0_4e40_892b_732be072817e.slice/crio-bf0a4305a38613a4269b9e0ef208362263264e20a44a9b445f4bc59ce513abe0 WatchSource:0}: Error finding container bf0a4305a38613a4269b9e0ef208362263264e20a44a9b445f4bc59ce513abe0: Status 404 returned error can't find the container with id bf0a4305a38613a4269b9e0ef208362263264e20a44a9b445f4bc59ce513abe0 Oct 30 00:59:18.476983 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:18.476954 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv"] Oct 30 00:59:18.479946 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:18.479911 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:18.480070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:18.479990 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:18.480070 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:18.480036 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:19.037596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:19.037007 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" event={"ID":"34591e8b-c9d0-4e40-892b-732be072817e","Type":"ContainerStarted","Data":"864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf"} Oct 30 00:59:19.037596 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:19.037076 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" event={"ID":"34591e8b-c9d0-4e40-892b-732be072817e","Type":"ContainerStarted","Data":"bf0a4305a38613a4269b9e0ef208362263264e20a44a9b445f4bc59ce513abe0"} Oct 30 00:59:19.137920 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:19.130736 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" podStartSLOduration=2.130714755 podStartE2EDuration="2.130714755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:19.113047165 +0000 UTC m=+1497.684973388" watchObservedRunningTime="2026-10-30 00:59:19.130714755 +0000 UTC m=+1497.702640981" Oct 30 00:59:19.908796 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:19.908760 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:19.915978 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:19.915952 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:20.096471 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:20.096437 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:20.117758 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:20.117721 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 00:59:21.906715 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:21.906684 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:59:21.907367 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:21.907343 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 00:59:23.520491 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.520459 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9"] Oct 30 00:59:23.539473 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.539416 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:23.548053 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.548021 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9"] Oct 30 00:59:23.557403 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.557371 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5f4f9b\"" Oct 30 00:59:23.581196 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.581162 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx"] Oct 30 00:59:23.588209 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.588181 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:23.606480 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.606453 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5f4f9b-dockercfg-wgr57\"" Oct 30 00:59:23.612224 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.612187 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx"] Oct 30 00:59:23.615374 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.615335 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d457c5a-4cc8-49e6-ac79-dafbd799684d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9\" (UID: \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:23.615553 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.615422 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdj2\" (UniqueName: \"kubernetes.io/projected/7d457c5a-4cc8-49e6-ac79-dafbd799684d-kube-api-access-brdj2\") pod \"payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9\" (UID: \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:23.821603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.813305 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a303900-2948-4ecd-8a56-04464434ee5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx\" (UID: \"1a303900-2948-4ecd-8a56-04464434ee5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:23.821603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.813809 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdwd\" (UniqueName: \"kubernetes.io/projected/1a303900-2948-4ecd-8a56-04464434ee5d-kube-api-access-2hdwd\") pod \"payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx\" (UID: \"1a303900-2948-4ecd-8a56-04464434ee5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:23.821603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.813995 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d457c5a-4cc8-49e6-ac79-dafbd799684d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9\" (UID: \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:23.821603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.814060 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdj2\" (UniqueName: \"kubernetes.io/projected/7d457c5a-4cc8-49e6-ac79-dafbd799684d-kube-api-access-brdj2\") pod \"payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9\" (UID: \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:23.821603 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.815416 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d457c5a-4cc8-49e6-ac79-dafbd799684d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9\" (UID: \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:23.883610 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.871499 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdj2\" (UniqueName: \"kubernetes.io/projected/7d457c5a-4cc8-49e6-ac79-dafbd799684d-kube-api-access-brdj2\") pod \"payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9\" (UID: \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:23.916072 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.916028 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdwd\" (UniqueName: \"kubernetes.io/projected/1a303900-2948-4ecd-8a56-04464434ee5d-kube-api-access-2hdwd\") pod \"payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx\" (UID: \"1a303900-2948-4ecd-8a56-04464434ee5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:23.924084 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.924050 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a303900-2948-4ecd-8a56-04464434ee5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx\" (UID: \"1a303900-2948-4ecd-8a56-04464434ee5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:23.925558 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.925379 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a303900-2948-4ecd-8a56-04464434ee5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx\" (UID: \"1a303900-2948-4ecd-8a56-04464434ee5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:24.005673 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:23.996323 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdwd\" (UniqueName: \"kubernetes.io/projected/1a303900-2948-4ecd-8a56-04464434ee5d-kube-api-access-2hdwd\") pod \"payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx\" (UID: \"1a303900-2948-4ecd-8a56-04464434ee5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:24.163560 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:24.163528 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:24.209375 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:24.209345 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:24.678719 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:59:24.678688 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a303900_2948_4ecd_8a56_04464434ee5d.slice/crio-f8acf455184b2b5fea4e77614dae04d4f391fff5876dcce47f97ab6be7a2afc6 WatchSource:0}: Error finding container f8acf455184b2b5fea4e77614dae04d4f391fff5876dcce47f97ab6be7a2afc6: Status 404 returned error can't find the container with id f8acf455184b2b5fea4e77614dae04d4f391fff5876dcce47f97ab6be7a2afc6 Oct 30 00:59:24.680551 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:24.680526 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx"] Oct 30 00:59:24.693324 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:24.693300 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9"] Oct 30 00:59:24.695479 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:59:24.695453 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d457c5a_4cc8_49e6_ac79_dafbd799684d.slice/crio-330f7b49b60af28131f97d17a1bc5d622b95614e6b30f30b9bd079e12c1806ff WatchSource:0}: Error finding container 330f7b49b60af28131f97d17a1bc5d622b95614e6b30f30b9bd079e12c1806ff: Status 404 returned error can't find the container with id 330f7b49b60af28131f97d17a1bc5d622b95614e6b30f30b9bd079e12c1806ff Oct 30 00:59:25.135936 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.135896 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" event={"ID":"7d457c5a-4cc8-49e6-ac79-dafbd799684d","Type":"ContainerStarted","Data":"b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d"} Oct 30 00:59:25.136166 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.135965 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" event={"ID":"7d457c5a-4cc8-49e6-ac79-dafbd799684d","Type":"ContainerStarted","Data":"330f7b49b60af28131f97d17a1bc5d622b95614e6b30f30b9bd079e12c1806ff"} Oct 30 00:59:25.136166 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.136016 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:25.137798 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.137773 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" event={"ID":"1a303900-2948-4ecd-8a56-04464434ee5d","Type":"ContainerStarted","Data":"d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3"} Oct 30 00:59:25.137942 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.137821 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" event={"ID":"1a303900-2948-4ecd-8a56-04464434ee5d","Type":"ContainerStarted","Data":"f8acf455184b2b5fea4e77614dae04d4f391fff5876dcce47f97ab6be7a2afc6"} Oct 30 00:59:25.137942 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.137851 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:25.179321 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.179265 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" podStartSLOduration=2.179251082 podStartE2EDuration="2.179251082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:25.17339246 +0000 UTC m=+1503.745318683" watchObservedRunningTime="2026-10-30 00:59:25.179251082 +0000 UTC m=+1503.751177335" Oct 30 00:59:25.214932 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.214879 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" podStartSLOduration=2.214864362 podStartE2EDuration="2.214864362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:25.211309284 +0000 UTC m=+1503.783235531" watchObservedRunningTime="2026-10-30 00:59:25.214864362 +0000 UTC m=+1503.786790584" Oct 30 00:59:25.441164 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.441048 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8979-1ce4d1"] Oct 30 00:59:25.452244 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.452212 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8979-1ce4d1" Oct 30 00:59:25.461425 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.461395 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8979-1ce4d1"] Oct 30 00:59:25.467888 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.467864 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v4hbx\"" Oct 30 00:59:25.620114 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.620079 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hwj\" (UniqueName: \"kubernetes.io/projected/49375f75-5b1a-43f5-8a47-24b773e0c939-kube-api-access-z7hwj\") pod \"test-curl-8979-1ce4d1\" (UID: \"49375f75-5b1a-43f5-8a47-24b773e0c939\") " pod="opendatahub/test-curl-8979-1ce4d1" Oct 30 00:59:25.721697 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.721605 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hwj\" (UniqueName: \"kubernetes.io/projected/49375f75-5b1a-43f5-8a47-24b773e0c939-kube-api-access-z7hwj\") pod \"test-curl-8979-1ce4d1\" (UID: \"49375f75-5b1a-43f5-8a47-24b773e0c939\") " pod="opendatahub/test-curl-8979-1ce4d1" Oct 30 00:59:25.745093 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.745064 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hwj\" (UniqueName: \"kubernetes.io/projected/49375f75-5b1a-43f5-8a47-24b773e0c939-kube-api-access-z7hwj\") pod \"test-curl-8979-1ce4d1\" (UID: \"49375f75-5b1a-43f5-8a47-24b773e0c939\") " pod="opendatahub/test-curl-8979-1ce4d1" Oct 30 00:59:25.770659 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:25.770628 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8979-1ce4d1" Oct 30 00:59:26.147614 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:59:26.147530 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49375f75_5b1a_43f5_8a47_24b773e0c939.slice/crio-77050b0fa9a379e1dd1ab2ec12b083dc8000edc060bf67a31aa6ee687a658e78 WatchSource:0}: Error finding container 77050b0fa9a379e1dd1ab2ec12b083dc8000edc060bf67a31aa6ee687a658e78: Status 404 returned error can't find the container with id 77050b0fa9a379e1dd1ab2ec12b083dc8000edc060bf67a31aa6ee687a658e78 Oct 30 00:59:26.149440 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:26.149415 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8979-1ce4d1"] Oct 30 00:59:27.169766 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:27.169729 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8979-1ce4d1" event={"ID":"49375f75-5b1a-43f5-8a47-24b773e0c939","Type":"ContainerStarted","Data":"72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219"} Oct 30 00:59:27.170167 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:27.169775 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8979-1ce4d1" event={"ID":"49375f75-5b1a-43f5-8a47-24b773e0c939","Type":"ContainerStarted","Data":"77050b0fa9a379e1dd1ab2ec12b083dc8000edc060bf67a31aa6ee687a658e78"} Oct 30 00:59:27.216442 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:27.216388 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8979-1ce4d1" podStartSLOduration=2.216372885 podStartE2EDuration="2.216372885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:27.210432703 +0000 UTC m=+1505.782358925" watchObservedRunningTime="2026-10-30 00:59:27.216372885 +0000 UTC m=+1505.788299107" Oct 30 00:59:28.441830 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:28.441792 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8979-1ce4d1"] Oct 30 00:59:28.447880 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:28.447851 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8979-1ce4d1"] Oct 30 00:59:29.074125 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.074082 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv"] Oct 30 00:59:29.075296 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.075263 4927 cpu_manager.go:414] "containerMap: removing container" podUID="49375f75-5b1a-43f5-8a47-24b773e0c939" containerName="test-curl-8979-1ce4d1" Oct 30 00:59:29.075296 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.075291 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49375f75-5b1a-43f5-8a47-24b773e0c939" containerName="test-curl-8979-1ce4d1" Oct 30 00:59:29.075542 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.075504 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="49375f75-5b1a-43f5-8a47-24b773e0c939" containerName="test-curl-8979-1ce4d1" Oct 30 00:59:29.080687 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.080661 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.105190 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.105164 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv"] Oct 30 00:59:29.108092 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.108065 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-43c02d-openshift-default-dockercfg-q4fqr\"" Oct 30 00:59:29.213658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.211843 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-token\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.213658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.211938 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-credential-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.213658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.212102 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-certs\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.213658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.212128 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-podinfo\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.213658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.212166 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.213658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.212239 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrrn\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-kube-api-access-jkrrn\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.213658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.212291 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-envoy\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.213658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.212330 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istiod-ca-cert\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.213658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.212444 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-data\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.258594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.255149 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8979-1ce4d1" podUID="49375f75-5b1a-43f5-8a47-24b773e0c939" containerName="test-curl-8979-1ce4d1" containerID="cri-o://72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219" gracePeriod=2 Oct 30 00:59:29.269896 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:59:29.269806 4927 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8979-1ce4d1\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="49375f75-5b1a-43f5-8a47-24b773e0c939" pod="opendatahub/test-curl-8979-1ce4d1" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.315676 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-data\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.315858 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-token\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.315915 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-credential-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.316076 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-certs\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.316107 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-podinfo\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.316143 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.316212 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrrn\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-kube-api-access-jkrrn\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.316264 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-envoy\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.316308 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istiod-ca-cert\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.317529 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.317328 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istiod-ca-cert\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.327658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.319317 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-credential-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.327658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.319905 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.327658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.319961 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-certs\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.327658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.320021 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-data\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.327658 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.322590 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-envoy\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.339594 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.331631 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-podinfo\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.358858 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.358665 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrrn\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-kube-api-access-jkrrn\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.363091 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.362759 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-token\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.412670 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.409921 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:29.804633 ip-10-0-139-66 kubenswrapper[4927]: W1030 00:59:29.804558 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2292a1de_9dcc_4d12_bf6c_8334bc4a8b6a.slice/crio-faab267407cf1d6acf3622f7d558f8c3395dff7d648edeafe7c784e8c1f5e3b0 WatchSource:0}: Error finding container faab267407cf1d6acf3622f7d558f8c3395dff7d648edeafe7c784e8c1f5e3b0: Status 404 returned error can't find the container with id faab267407cf1d6acf3622f7d558f8c3395dff7d648edeafe7c784e8c1f5e3b0 Oct 30 00:59:29.804999 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.804974 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv"] Oct 30 00:59:29.808042 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.808004 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:29.808152 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.808073 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:29.808152 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:29.808126 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:30.266282 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:30.266242 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" event={"ID":"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a","Type":"ContainerStarted","Data":"4e5185d39cc1e4e1df84d34d9df7b2d44231be8aa577532b9a7208538475eee1"} Oct 30 00:59:30.266466 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:30.266310 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" event={"ID":"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a","Type":"ContainerStarted","Data":"faab267407cf1d6acf3622f7d558f8c3395dff7d648edeafe7c784e8c1f5e3b0"} Oct 30 00:59:30.314557 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:30.314504 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podStartSLOduration=1.314488119 podStartE2EDuration="1.314488119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:30.310966582 +0000 UTC m=+1508.882892803" watchObservedRunningTime="2026-10-30 00:59:30.314488119 +0000 UTC m=+1508.886414341" Oct 30 00:59:30.410802 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:30.410771 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:30.416773 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:30.416745 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:31.274647 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:31.274611 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:31.275519 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:31.275498 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 00:59:31.540745 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:31.540682 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8979-1ce4d1" Oct 30 00:59:31.672737 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:31.672701 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49375f75-5b1a-43f5-8a47-24b773e0c939-kube-api-access-z7hwj\" (UniqueName: \"kubernetes.io/projected/49375f75-5b1a-43f5-8a47-24b773e0c939-kube-api-access-z7hwj\") pod \"49375f75-5b1a-43f5-8a47-24b773e0c939\" (UID: \"49375f75-5b1a-43f5-8a47-24b773e0c939\") " Oct 30 00:59:31.674932 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:31.674906 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49375f75-5b1a-43f5-8a47-24b773e0c939-kube-api-access-z7hwj" pod "49375f75-5b1a-43f5-8a47-24b773e0c939" (UID: "49375f75-5b1a-43f5-8a47-24b773e0c939"). InnerVolumeSpecName "kube-api-access-z7hwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:31.774672 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:31.774639 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7hwj\" (UniqueName: \"kubernetes.io/projected/49375f75-5b1a-43f5-8a47-24b773e0c939-kube-api-access-z7hwj\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:59:32.380289 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:32.380246 4927 generic.go:356] "Generic (PLEG): container finished" podID="49375f75-5b1a-43f5-8a47-24b773e0c939" containerID="72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219" exitCode=137 Oct 30 00:59:32.380925 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:32.380794 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8979-1ce4d1" Oct 30 00:59:32.381247 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:32.381222 4927 scope.go:122] "RemoveContainer" containerID="72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219" Oct 30 00:59:32.421296 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:32.421268 4927 scope.go:122] "RemoveContainer" containerID="72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219" Oct 30 00:59:32.421781 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:59:32.421748 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219\": container with ID starting with 72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219 not found: ID does not exist" containerID="72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219" Oct 30 00:59:32.421880 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:32.421793 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219"} err="failed to get container status \"72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219\": rpc error: code = NotFound desc = could not find container \"72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219\": container with ID starting with 72c5cc547d7189c8179755b9d92b99d93381affd610951ca543807079945c219 not found: ID does not exist" Oct 30 00:59:33.574866 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:33.574827 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49375f75-5b1a-43f5-8a47-24b773e0c939" path="/var/lib/kubelet/pods/49375f75-5b1a-43f5-8a47-24b773e0c939/volumes" Oct 30 00:59:34.554965 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:59:34.554921 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 00:59:36.151980 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:36.151949 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 00:59:36.152435 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:36.152034 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 00:59:51.382671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:51.381423 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44"] Oct 30 00:59:51.382671 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:51.381754 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" podUID="685d1eb5-c754-4840-93af-583859a64759" containerName="maas-api" containerID="cri-o://1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0" gracePeriod=30 Oct 30 00:59:52.593838 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.593810 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:59:52.696698 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.696662 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-metrics-certs\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-metrics-certs\") pod \"685d1eb5-c754-4840-93af-583859a64759\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " Oct 30 00:59:52.696872 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.696844 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/685d1eb5-c754-4840-93af-583859a64759-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/685d1eb5-c754-4840-93af-583859a64759-openshift-service-ca\") pod \"685d1eb5-c754-4840-93af-583859a64759\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " Oct 30 00:59:52.696926 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.696914 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685d1eb5-c754-4840-93af-583859a64759-kube-api-access-v745f\" (UniqueName: \"kubernetes.io/projected/685d1eb5-c754-4840-93af-583859a64759-kube-api-access-v745f\") pod \"685d1eb5-c754-4840-93af-583859a64759\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " Oct 30 00:59:52.696974 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.696953 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-maas-api-tls\") pod \"685d1eb5-c754-4840-93af-583859a64759\" (UID: \"685d1eb5-c754-4840-93af-583859a64759\") " Oct 30 00:59:52.697241 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.697212 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685d1eb5-c754-4840-93af-583859a64759-openshift-service-ca" pod "685d1eb5-c754-4840-93af-583859a64759" (UID: "685d1eb5-c754-4840-93af-583859a64759"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:52.698269 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.698245 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/685d1eb5-c754-4840-93af-583859a64759-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:59:52.699184 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.699153 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-maas-api-tls" pod "685d1eb5-c754-4840-93af-583859a64759" (UID: "685d1eb5-c754-4840-93af-583859a64759"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:52.699291 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.699245 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-metrics-certs" pod "685d1eb5-c754-4840-93af-583859a64759" (UID: "685d1eb5-c754-4840-93af-583859a64759"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:59:52.699291 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.699275 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685d1eb5-c754-4840-93af-583859a64759-kube-api-access-v745f" pod "685d1eb5-c754-4840-93af-583859a64759" (UID: "685d1eb5-c754-4840-93af-583859a64759"). InnerVolumeSpecName "kube-api-access-v745f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:52.800180 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.800092 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v745f\" (UniqueName: \"kubernetes.io/projected/685d1eb5-c754-4840-93af-583859a64759-kube-api-access-v745f\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:59:52.800180 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.800124 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:59:52.800180 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:52.800134 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/685d1eb5-c754-4840-93af-583859a64759-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 00:59:53.364212 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.364179 4927 generic.go:356] "Generic (PLEG): container finished" podID="685d1eb5-c754-4840-93af-583859a64759" containerID="1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0" exitCode=0 Oct 30 00:59:53.364423 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.364234 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" Oct 30 00:59:53.364423 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.364259 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" event={"ID":"685d1eb5-c754-4840-93af-583859a64759","Type":"ContainerDied","Data":"1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0"} Oct 30 00:59:53.364423 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.364320 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44" event={"ID":"685d1eb5-c754-4840-93af-583859a64759","Type":"ContainerDied","Data":"354a0f8b19c4d9349fa34da180185135d0bdd919af3235a2b1c935c349d4470c"} Oct 30 00:59:53.364423 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.364351 4927 scope.go:122] "RemoveContainer" containerID="1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0" Oct 30 00:59:53.376805 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.376781 4927 scope.go:122] "RemoveContainer" containerID="1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0" Oct 30 00:59:53.377105 ip-10-0-139-66 kubenswrapper[4927]: E1030 00:59:53.377084 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0\": container with ID starting with 1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0 not found: ID does not exist" containerID="1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0" Oct 30 00:59:53.377182 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.377117 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0"} err="failed to get container status \"1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0\": rpc error: code = NotFound desc = could not find container \"1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0\": container with ID starting with 1453ecc0c62c7e8cfa246aa509b42522a9eb2a930b8d7e4c451e0d95d04b27f0 not found: ID does not exist" Oct 30 00:59:53.417621 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.417588 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44"] Oct 30 00:59:53.428444 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.428416 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8090513-784b489945-6tf44"] Oct 30 00:59:53.573635 ip-10-0-139-66 kubenswrapper[4927]: I1030 00:59:53.573602 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="685d1eb5-c754-4840-93af-583859a64759" path="/var/lib/kubelet/pods/685d1eb5-c754-4840-93af-583859a64759/volumes" Oct 30 01:00:04.445540 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:00:04.445486 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:00:12.685844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:12.685803 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8"] Oct 30 01:00:12.686538 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:12.686167 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" containerID="cri-o://6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69" gracePeriod=30 Oct 30 01:00:12.698021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:12.697987 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55"] Oct 30 01:00:12.698284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:12.698256 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" containerID="cri-o://30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48" gracePeriod=30 Oct 30 01:00:18.022294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.022263 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 01:00:18.078954 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.078924 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 01:00:18.116608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.089474 4927 generic.go:356] "Generic (PLEG): container finished" podID="81ef110e-b36f-42e1-a30c-399f16249151" containerID="30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48" exitCode=0 Oct 30 01:00:18.116608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.089845 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" event={"ID":"81ef110e-b36f-42e1-a30c-399f16249151","Type":"ContainerDied","Data":"30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48"} Oct 30 01:00:18.116608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.089898 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" event={"ID":"81ef110e-b36f-42e1-a30c-399f16249151","Type":"ContainerDied","Data":"27aee5166bfeb5ce8838bee399be90247101dcd59b0d8ee83dfacfa970caae86"} Oct 30 01:00:18.116608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.089927 4927 scope.go:122] "RemoveContainer" containerID="30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48" Oct 30 01:00:18.116608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.093312 4927 generic.go:356] "Generic (PLEG): container finished" podID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerID="6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69" exitCode=0 Oct 30 01:00:18.116608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.093430 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" event={"ID":"bdb8a0de-60b0-42d8-bb7a-1e83e81db566","Type":"ContainerDied","Data":"6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69"} Oct 30 01:00:18.116608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.093490 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" event={"ID":"bdb8a0de-60b0-42d8-bb7a-1e83e81db566","Type":"ContainerDied","Data":"1c3db6c490a87200742f5a3be84dd6c1ee0c5d4ec5a19564d490fc7ecf2cf99c"} Oct 30 01:00:18.116608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.093678 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8" Oct 30 01:00:18.132817 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.132794 4927 scope.go:122] "RemoveContainer" containerID="30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48" Oct 30 01:00:18.133244 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:00:18.133213 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48\": container with ID starting with 30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48 not found: ID does not exist" containerID="30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48" Oct 30 01:00:18.133347 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.133258 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48"} err="failed to get container status \"30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48\": rpc error: code = NotFound desc = could not find container \"30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48\": container with ID starting with 30e995c271c8afba8b693974b03df517019f52fd5ba8ae0720c9159886514d48 not found: ID does not exist" Oct 30 01:00:18.133347 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.133286 4927 scope.go:122] "RemoveContainer" containerID="6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69" Oct 30 01:00:18.135647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.135620 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-kube-api-access-p4bmn\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-kube-api-access-p4bmn\") pod \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " Oct 30 01:00:18.135762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.135700 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-token\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-token\") pod \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " Oct 30 01:00:18.135762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.135754 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-certs\") pod \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " Oct 30 01:00:18.135961 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.135798 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-podinfo\") pod \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " Oct 30 01:00:18.135961 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.135870 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istiod-ca-cert\") pod \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " Oct 30 01:00:18.136070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.135997 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-data\") pod \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " Oct 30 01:00:18.136161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.136142 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-envoy\") pod \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " Oct 30 01:00:18.136280 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.136259 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-socket\") pod \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " Oct 30 01:00:18.136354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.136300 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-credential-socket\") pod \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\" (UID: \"bdb8a0de-60b0-42d8-bb7a-1e83e81db566\") " Oct 30 01:00:18.138193 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.137410 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-credential-socket" pod "bdb8a0de-60b0-42d8-bb7a-1e83e81db566" (UID: "bdb8a0de-60b0-42d8-bb7a-1e83e81db566"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.138472 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.138447 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-certs" pod "bdb8a0de-60b0-42d8-bb7a-1e83e81db566" (UID: "bdb8a0de-60b0-42d8-bb7a-1e83e81db566"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.141654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.139230 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istiod-ca-cert" pod "bdb8a0de-60b0-42d8-bb7a-1e83e81db566" (UID: "bdb8a0de-60b0-42d8-bb7a-1e83e81db566"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:18.141654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.139586 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-socket" pod "bdb8a0de-60b0-42d8-bb7a-1e83e81db566" (UID: "bdb8a0de-60b0-42d8-bb7a-1e83e81db566"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.141654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.140925 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-data" pod "bdb8a0de-60b0-42d8-bb7a-1e83e81db566" (UID: "bdb8a0de-60b0-42d8-bb7a-1e83e81db566"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.147923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.147274 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-kube-api-access-p4bmn" pod "bdb8a0de-60b0-42d8-bb7a-1e83e81db566" (UID: "bdb8a0de-60b0-42d8-bb7a-1e83e81db566"). InnerVolumeSpecName "kube-api-access-p4bmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:18.147923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.147732 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-podinfo" pod "bdb8a0de-60b0-42d8-bb7a-1e83e81db566" (UID: "bdb8a0de-60b0-42d8-bb7a-1e83e81db566"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:00:18.148659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.148634 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-token" pod "bdb8a0de-60b0-42d8-bb7a-1e83e81db566" (UID: "bdb8a0de-60b0-42d8-bb7a-1e83e81db566"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:18.148884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.148868 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-envoy" pod "bdb8a0de-60b0-42d8-bb7a-1e83e81db566" (UID: "bdb8a0de-60b0-42d8-bb7a-1e83e81db566"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.155243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.155223 4927 scope.go:122] "RemoveContainer" containerID="6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69" Oct 30 01:00:18.155784 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:00:18.155749 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69\": container with ID starting with 6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69 not found: ID does not exist" containerID="6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69" Oct 30 01:00:18.155893 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.155792 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69"} err="failed to get container status \"6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69\": rpc error: code = NotFound desc = could not find container \"6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69\": container with ID starting with 6604e1ca9c84e50eb7c8c634b73d7220c0e9973d55660f9c515d5cdbfc4b0f69 not found: ID does not exist" Oct 30 01:00:18.237852 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.237766 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-envoy\") pod \"81ef110e-b36f-42e1-a30c-399f16249151\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " Oct 30 01:00:18.238001 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.237881 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-certs\") pod \"81ef110e-b36f-42e1-a30c-399f16249151\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " Oct 30 01:00:18.238001 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.237918 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81ef110e-b36f-42e1-a30c-399f16249151-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ef110e-b36f-42e1-a30c-399f16249151-istio-podinfo\") pod \"81ef110e-b36f-42e1-a30c-399f16249151\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " Oct 30 01:00:18.238083 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.238017 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-data\") pod \"81ef110e-b36f-42e1-a30c-399f16249151\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " Oct 30 01:00:18.238118 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.238096 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ef110e-b36f-42e1-a30c-399f16249151-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ef110e-b36f-42e1-a30c-399f16249151-istiod-ca-cert\") pod \"81ef110e-b36f-42e1-a30c-399f16249151\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " Oct 30 01:00:18.238202 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.238186 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-socket\") pod \"81ef110e-b36f-42e1-a30c-399f16249151\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " Oct 30 01:00:18.238261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.238248 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-istio-token\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-istio-token\") pod \"81ef110e-b36f-42e1-a30c-399f16249151\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " Oct 30 01:00:18.238305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.238291 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-kube-api-access-xdxl4\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-kube-api-access-xdxl4\") pod \"81ef110e-b36f-42e1-a30c-399f16249151\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " Oct 30 01:00:18.238391 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.238378 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-credential-socket\") pod \"81ef110e-b36f-42e1-a30c-399f16249151\" (UID: \"81ef110e-b36f-42e1-a30c-399f16249151\") " Oct 30 01:00:18.239273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239243 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.239273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239273 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.239481 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239289 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.239481 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239304 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bmn\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-kube-api-access-p4bmn\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.239481 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239318 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.239481 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239331 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.239481 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239350 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.239481 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239363 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.239481 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239375 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb8a0de-60b0-42d8-bb7a-1e83e81db566-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.239851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.239644 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-credential-socket" pod "81ef110e-b36f-42e1-a30c-399f16249151" (UID: "81ef110e-b36f-42e1-a30c-399f16249151"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.240052 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.240025 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ef110e-b36f-42e1-a30c-399f16249151-istiod-ca-cert" pod "81ef110e-b36f-42e1-a30c-399f16249151" (UID: "81ef110e-b36f-42e1-a30c-399f16249151"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:18.240252 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.240234 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-certs" pod "81ef110e-b36f-42e1-a30c-399f16249151" (UID: "81ef110e-b36f-42e1-a30c-399f16249151"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.240905 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.240876 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-data" pod "81ef110e-b36f-42e1-a30c-399f16249151" (UID: "81ef110e-b36f-42e1-a30c-399f16249151"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.241004 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.240951 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-socket" pod "81ef110e-b36f-42e1-a30c-399f16249151" (UID: "81ef110e-b36f-42e1-a30c-399f16249151"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.242266 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.242238 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-envoy" pod "81ef110e-b36f-42e1-a30c-399f16249151" (UID: "81ef110e-b36f-42e1-a30c-399f16249151"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:18.242431 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.242411 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81ef110e-b36f-42e1-a30c-399f16249151-istio-podinfo" pod "81ef110e-b36f-42e1-a30c-399f16249151" (UID: "81ef110e-b36f-42e1-a30c-399f16249151"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:00:18.242922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.242899 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-istio-token" pod "81ef110e-b36f-42e1-a30c-399f16249151" (UID: "81ef110e-b36f-42e1-a30c-399f16249151"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:18.243381 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.243359 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-kube-api-access-xdxl4" pod "81ef110e-b36f-42e1-a30c-399f16249151" (UID: "81ef110e-b36f-42e1-a30c-399f16249151"). InnerVolumeSpecName "kube-api-access-xdxl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:18.341673 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.341638 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.341673 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.341667 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.341673 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.341676 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ef110e-b36f-42e1-a30c-399f16249151-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.341673 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.341684 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.341944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.341691 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ef110e-b36f-42e1-a30c-399f16249151-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.341944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.341700 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.341944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.341709 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.341944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.341719 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdxl4\" (UniqueName: \"kubernetes.io/projected/81ef110e-b36f-42e1-a30c-399f16249151-kube-api-access-xdxl4\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.341944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.341728 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ef110e-b36f-42e1-a30c-399f16249151-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:18.442793 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.442747 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8"] Oct 30 01:00:18.453845 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:18.453816 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-8rhg8"] Oct 30 01:00:19.103921 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:19.103839 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55" Oct 30 01:00:19.158680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:19.158649 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55"] Oct 30 01:00:19.171421 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:19.171394 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xfr55"] Oct 30 01:00:19.578944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:19.578902 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ef110e-b36f-42e1-a30c-399f16249151" path="/var/lib/kubelet/pods/81ef110e-b36f-42e1-a30c-399f16249151/volumes" Oct 30 01:00:19.579611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:19.579548 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" path="/var/lib/kubelet/pods/bdb8a0de-60b0-42d8-bb7a-1e83e81db566/volumes" Oct 30 01:00:34.846856 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:00:34.846811 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:00:42.037802 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:42.037761 4927 generic.go:356] "Generic (PLEG): container finished" podID="34591e8b-c9d0-4e40-892b-732be072817e" containerID="864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf" exitCode=0 Oct 30 01:00:42.038349 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:42.037821 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" event={"ID":"34591e8b-c9d0-4e40-892b-732be072817e","Type":"ContainerDied","Data":"864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf"} Oct 30 01:00:42.038464 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:42.038444 4927 scope.go:122] "RemoveContainer" containerID="864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf" Oct 30 01:00:42.040563 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:42.040530 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:42.040681 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:42.040623 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:42.040681 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:42.040662 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:43.092592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.086699 4927 generic.go:356] "Generic (PLEG): container finished" podID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerID="4e5185d39cc1e4e1df84d34d9df7b2d44231be8aa577532b9a7208538475eee1" exitCode=0 Oct 30 01:00:43.092592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.086934 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" event={"ID":"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a","Type":"ContainerDied","Data":"4e5185d39cc1e4e1df84d34d9df7b2d44231be8aa577532b9a7208538475eee1"} Oct 30 01:00:43.092592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.087481 4927 scope.go:122] "RemoveContainer" containerID="4e5185d39cc1e4e1df84d34d9df7b2d44231be8aa577532b9a7208538475eee1" Oct 30 01:00:43.106066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.105999 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:43.106230 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.106130 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:43.106230 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.106181 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:43.155653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.155547 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" event={"ID":"34591e8b-c9d0-4e40-892b-732be072817e","Type":"ContainerStarted","Data":"b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07"} Oct 30 01:00:43.908949 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.908914 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 01:00:43.910136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.910103 4927 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:00:43.910304 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:43.910178 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 01:00:44.167297 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:44.167209 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" event={"ID":"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a","Type":"ContainerStarted","Data":"6d1b139e4634327f1f4ccf494c5f3182f858e448eb8ec7fff9e74576fc6513c8"} Oct 30 01:00:44.412596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:44.411941 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 01:00:44.424617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:44.422700 4927 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:00:44.424617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:44.422769 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 01:00:44.909597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:44.908943 4927 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:00:44.909597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:44.909010 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 01:00:45.434595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:45.429968 4927 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:00:45.434595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:45.430036 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 01:00:45.916595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:45.910964 4927 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:00:45.916595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:45.911048 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 01:00:46.287643 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:46.276665 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 01:00:46.415599 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:46.412927 4927 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:00:46.415599 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:46.412988 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 01:00:46.672644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:46.672255 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5"] Oct 30 01:00:46.672644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:46.672482 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" containerID="cri-o://291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855" gracePeriod=30 Oct 30 01:00:46.951652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:46.933918 4927 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:00:46.951652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:46.933980 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 01:00:47.437599 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:47.437451 4927 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:00:47.437599 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:47.437514 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 01:00:47.910208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:47.910168 4927 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:00:47.910869 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:47.910832 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 01:00:48.416595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:48.416086 4927 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:00:48.416595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:48.416146 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 01:00:48.912373 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:48.912335 4927 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:00:48.912976 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:48.912946 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 01:00:49.413216 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:49.413176 4927 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:00:49.413561 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:49.413526 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 01:00:49.915271 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:49.915239 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 01:00:49.916238 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:49.916208 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 01:00:49.917466 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:49.917439 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 01:00:50.411315 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:50.411278 4927 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:00:50.411520 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:50.411335 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 01:00:51.416407 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:51.416374 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 01:00:51.417655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:51.417630 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 01:00:51.934807 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:51.934781 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 01:00:52.060916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.054310 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-socket\") pod \"43321879-3683-44ec-b6b6-11820f4ad9f2\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " Oct 30 01:00:52.060916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.054372 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-podinfo\") pod \"43321879-3683-44ec-b6b6-11820f4ad9f2\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " Oct 30 01:00:52.060916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.054416 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-token\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-token\") pod \"43321879-3683-44ec-b6b6-11820f4ad9f2\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " Oct 30 01:00:52.060916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.054483 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-certs\") pod \"43321879-3683-44ec-b6b6-11820f4ad9f2\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " Oct 30 01:00:52.060916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.054527 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-kube-api-access-pp8pd\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-kube-api-access-pp8pd\") pod \"43321879-3683-44ec-b6b6-11820f4ad9f2\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " Oct 30 01:00:52.060916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.054695 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43321879-3683-44ec-b6b6-11820f4ad9f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43321879-3683-44ec-b6b6-11820f4ad9f2-istiod-ca-cert\") pod \"43321879-3683-44ec-b6b6-11820f4ad9f2\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " Oct 30 01:00:52.060916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.054785 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-credential-socket\") pod \"43321879-3683-44ec-b6b6-11820f4ad9f2\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " Oct 30 01:00:52.060916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.054850 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-data\") pod \"43321879-3683-44ec-b6b6-11820f4ad9f2\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " Oct 30 01:00:52.060916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.055013 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-envoy\") pod \"43321879-3683-44ec-b6b6-11820f4ad9f2\" (UID: \"43321879-3683-44ec-b6b6-11820f4ad9f2\") " Oct 30 01:00:52.094950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.088058 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-socket" pod "43321879-3683-44ec-b6b6-11820f4ad9f2" (UID: "43321879-3683-44ec-b6b6-11820f4ad9f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:52.094950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.091869 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-credential-socket" pod "43321879-3683-44ec-b6b6-11820f4ad9f2" (UID: "43321879-3683-44ec-b6b6-11820f4ad9f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:52.094950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.091942 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-data" pod "43321879-3683-44ec-b6b6-11820f4ad9f2" (UID: "43321879-3683-44ec-b6b6-11820f4ad9f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:52.094950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.093313 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:52.094950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.093354 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:52.094950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.093370 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:52.144595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.103001 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-certs" pod "43321879-3683-44ec-b6b6-11820f4ad9f2" (UID: "43321879-3683-44ec-b6b6-11820f4ad9f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:52.144595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.103186 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43321879-3683-44ec-b6b6-11820f4ad9f2-istiod-ca-cert" pod "43321879-3683-44ec-b6b6-11820f4ad9f2" (UID: "43321879-3683-44ec-b6b6-11820f4ad9f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:52.274595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.273266 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-token" pod "43321879-3683-44ec-b6b6-11820f4ad9f2" (UID: "43321879-3683-44ec-b6b6-11820f4ad9f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:52.274595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.273385 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-envoy" pod "43321879-3683-44ec-b6b6-11820f4ad9f2" (UID: "43321879-3683-44ec-b6b6-11820f4ad9f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:52.291696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.291664 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:52.291959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.291939 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43321879-3683-44ec-b6b6-11820f4ad9f2-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:52.293554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.293495 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-podinfo" pod "43321879-3683-44ec-b6b6-11820f4ad9f2" (UID: "43321879-3683-44ec-b6b6-11820f4ad9f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:00:52.296248 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.296167 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-kube-api-access-pp8pd" pod "43321879-3683-44ec-b6b6-11820f4ad9f2" (UID: "43321879-3683-44ec-b6b6-11820f4ad9f2"). InnerVolumeSpecName "kube-api-access-pp8pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:52.421868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.413307 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:52.423357 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.423327 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:52.423523 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.423510 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:52.423653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.423640 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8pd\" (UniqueName: \"kubernetes.io/projected/43321879-3683-44ec-b6b6-11820f4ad9f2-kube-api-access-pp8pd\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:00:52.434254 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.434227 4927 generic.go:356] "Generic (PLEG): container finished" podID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerID="291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855" exitCode=0 Oct 30 01:00:52.434718 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.434689 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" event={"ID":"43321879-3683-44ec-b6b6-11820f4ad9f2","Type":"ContainerDied","Data":"291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855"} Oct 30 01:00:52.434845 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.434778 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" event={"ID":"43321879-3683-44ec-b6b6-11820f4ad9f2","Type":"ContainerDied","Data":"e0a7b5999710d181132816824e9e89d5f77fd1143817b01afc07d8c0cbd80a1b"} Oct 30 01:00:52.434845 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.434817 4927 scope.go:122] "RemoveContainer" containerID="291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855" Oct 30 01:00:52.435534 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.435477 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5" Oct 30 01:00:52.453837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.453815 4927 scope.go:122] "RemoveContainer" containerID="11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f" Oct 30 01:00:52.464820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.464802 4927 scope.go:122] "RemoveContainer" containerID="291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855" Oct 30 01:00:52.465102 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:00:52.465079 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855\": container with ID starting with 291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855 not found: ID does not exist" containerID="291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855" Oct 30 01:00:52.465157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.465114 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855"} err="failed to get container status \"291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855\": rpc error: code = NotFound desc = could not find container \"291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855\": container with ID starting with 291155bb750cd2b5ff85e91ee42b0a2ea0edc4e98396c994daea70981a6bb855 not found: ID does not exist" Oct 30 01:00:52.465157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.465131 4927 scope.go:122] "RemoveContainer" containerID="11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f" Oct 30 01:00:52.465403 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:00:52.465383 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f\": container with ID starting with 11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f not found: ID does not exist" containerID="11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f" Oct 30 01:00:52.465460 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.465407 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f"} err="failed to get container status \"11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f\": rpc error: code = NotFound desc = could not find container \"11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f\": container with ID starting with 11a3b94885a8821753a7a703aa9bdfc27ace149dfd67902f7c684c1d444eff9f not found: ID does not exist" Oct 30 01:00:52.495707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.495680 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5"] Oct 30 01:00:52.506292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:52.506268 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d8090513-openshift-default-6f56d54b45-xstc5"] Oct 30 01:00:53.311087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.311044 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6"] Oct 30 01:00:53.312239 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312209 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" Oct 30 01:00:53.312239 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312238 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" Oct 30 01:00:53.312458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312293 4927 cpu_manager.go:414] "containerMap: removing container" podUID="685d1eb5-c754-4840-93af-583859a64759" containerName="maas-api" Oct 30 01:00:53.312458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312302 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685d1eb5-c754-4840-93af-583859a64759" containerName="maas-api" Oct 30 01:00:53.312458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312321 4927 cpu_manager.go:414] "containerMap: removing container" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" Oct 30 01:00:53.312458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312329 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" Oct 30 01:00:53.312458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312394 4927 cpu_manager.go:414] "containerMap: removing container" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" Oct 30 01:00:53.312458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312402 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" Oct 30 01:00:53.312759 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312582 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" Oct 30 01:00:53.312759 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312597 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ef110e-b36f-42e1-a30c-399f16249151" containerName="istio-proxy" Oct 30 01:00:53.312759 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312607 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb8a0de-60b0-42d8-bb7a-1e83e81db566" containerName="istio-proxy" Oct 30 01:00:53.312759 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312622 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="685d1eb5-c754-4840-93af-583859a64759" containerName="maas-api" Oct 30 01:00:53.312759 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.312630 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" Oct 30 01:00:53.318174 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.318155 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.344219 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.344194 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-780077-serving-cert\"" Oct 30 01:00:53.361309 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.352045 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6"] Oct 30 01:00:53.443936 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.443904 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.444355 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.443961 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.444355 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.444193 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmg8\" (UniqueName: \"kubernetes.io/projected/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-kube-api-access-gxmg8\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.444355 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.444319 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-metrics-certs\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.546080 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.546049 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-metrics-certs\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.546257 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.546098 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.546257 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.546143 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.546403 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.546384 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmg8\" (UniqueName: \"kubernetes.io/projected/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-kube-api-access-gxmg8\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.546970 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.546941 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.548768 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.548745 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-metrics-certs\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.548844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.548745 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.578883 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.578801 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" path="/var/lib/kubelet/pods/43321879-3683-44ec-b6b6-11820f4ad9f2/volumes" Oct 30 01:00:53.600634 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.600598 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmg8\" (UniqueName: \"kubernetes.io/projected/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-kube-api-access-gxmg8\") pod \"maas-api-e2e-inf-a-780077-7f49667468-6znm6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:53.632923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:53.632887 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:54.036075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:54.036045 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6"] Oct 30 01:00:54.037071 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:00:54.037029 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a1e09b_20ef_4a61_833e_48e94dd4bbc6.slice/crio-22c479867d1a9fd38ea9fb81d6fbc2ce23aa6de9015678c5017c28f579b10033 WatchSource:0}: Error finding container 22c479867d1a9fd38ea9fb81d6fbc2ce23aa6de9015678c5017c28f579b10033: Status 404 returned error can't find the container with id 22c479867d1a9fd38ea9fb81d6fbc2ce23aa6de9015678c5017c28f579b10033 Oct 30 01:00:54.455494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:54.455461 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" event={"ID":"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6","Type":"ContainerStarted","Data":"3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293"} Oct 30 01:00:54.455976 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:54.455516 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" event={"ID":"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6","Type":"ContainerStarted","Data":"22c479867d1a9fd38ea9fb81d6fbc2ce23aa6de9015678c5017c28f579b10033"} Oct 30 01:00:54.455976 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:54.455606 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:00:54.503055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:54.502992 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" podStartSLOduration=1.200516757 podStartE2EDuration="1.502978186s" totalImagesPullingTime="302.461429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:54.49865762 +0000 UTC m=+1593.070583843" watchObservedRunningTime="2026-10-30 01:00:54.502978186 +0000 UTC m=+1593.074904408" Oct 30 01:00:59.781915 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.781877 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt"] Oct 30 01:00:59.783215 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.783189 4927 cpu_manager.go:414] "containerMap: removing container" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" Oct 30 01:00:59.783215 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.783210 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43321879-3683-44ec-b6b6-11820f4ad9f2" containerName="istio-proxy" Oct 30 01:00:59.788226 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.788193 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.794255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.794225 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.794389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.794284 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.794471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.794443 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnxd\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-kube-api-access-2lnxd\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.794531 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.794493 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.794726 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.794702 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.794808 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.794765 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.794890 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.794860 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.794956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.794924 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.795031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.795002 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060bfdb8-6cf7-403d-85d2-e81900dc442d-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.805186 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.805167 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-969b92-openshift-default-dockercfg-4fk78\"" Oct 30 01:00:59.828672 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.828643 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt"] Oct 30 01:00:59.915958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.911936 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnxd\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-kube-api-access-2lnxd\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.915958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.911998 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.915958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.912172 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.915958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.912216 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.975817 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.976164 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.976423 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.976487 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.976566 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.976625 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060bfdb8-6cf7-403d-85d2-e81900dc442d-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.976780 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:00:59.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:00:59.976834 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:00.024596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.018878 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:00.024596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.019175 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:00.024596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.021523 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060bfdb8-6cf7-403d-85d2-e81900dc442d-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:00.058665 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.058562 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnxd\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-kube-api-access-2lnxd\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:00.063591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.062141 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:00.075787 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.075754 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:00.113886 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.113852 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:00.489755 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.489723 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:01:00.612015 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:01:00.611961 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060bfdb8_6cf7_403d_85d2_e81900dc442d.slice/crio-40423cded3c5aeacd2519cac0cc00022ca91fcf84e23e026f62eb231a58257f4 WatchSource:0}: Error finding container 40423cded3c5aeacd2519cac0cc00022ca91fcf84e23e026f62eb231a58257f4: Status 404 returned error can't find the container with id 40423cded3c5aeacd2519cac0cc00022ca91fcf84e23e026f62eb231a58257f4 Oct 30 01:01:00.612149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.612015 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt"] Oct 30 01:01:00.616051 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.616008 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:00.616162 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.616097 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:00.616162 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:00.616146 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:01.541444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:01.541407 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" event={"ID":"060bfdb8-6cf7-403d-85d2-e81900dc442d","Type":"ContainerStarted","Data":"53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83"} Oct 30 01:01:01.542035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:01.541459 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" event={"ID":"060bfdb8-6cf7-403d-85d2-e81900dc442d","Type":"ContainerStarted","Data":"40423cded3c5aeacd2519cac0cc00022ca91fcf84e23e026f62eb231a58257f4"} Oct 30 01:01:01.618366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:01.618312 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" podStartSLOduration=2.618297846 podStartE2EDuration="2.618297846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:01.602750921 +0000 UTC m=+1600.174677142" watchObservedRunningTime="2026-10-30 01:01:01.618297846 +0000 UTC m=+1600.190224069" Oct 30 01:01:02.114894 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:02.114858 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:02.120659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:02.120634 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:02.552440 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:02.552405 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:02.553326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:02.553303 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:01:04.464910 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:01:04.464870 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:01:07.373855 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.373822 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt"] Oct 30 01:01:07.374324 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.374074 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" podUID="8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" containerName="maas-api" containerID="cri-o://54631015de9dcc7f738a3735878a157cad15c16024d535b5fa365819165ceb0f" gracePeriod=30 Oct 30 01:01:07.642028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.641953 4927 generic.go:356] "Generic (PLEG): container finished" podID="8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" containerID="54631015de9dcc7f738a3735878a157cad15c16024d535b5fa365819165ceb0f" exitCode=0 Oct 30 01:01:07.642183 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.642142 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" event={"ID":"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa","Type":"ContainerDied","Data":"54631015de9dcc7f738a3735878a157cad15c16024d535b5fa365819165ceb0f"} Oct 30 01:01:07.795602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.793077 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 01:01:07.906663 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.902536 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-metrics-certs\") pod \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " Oct 30 01:01:07.906663 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.902673 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-kube-api-access-bbq67\" (UniqueName: \"kubernetes.io/projected/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-kube-api-access-bbq67\") pod \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " Oct 30 01:01:07.906663 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.902798 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-openshift-service-ca\") pod \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " Oct 30 01:01:07.906663 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.902872 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-maas-api-tls\") pod \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\" (UID: \"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa\") " Oct 30 01:01:07.926174 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.925959 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-openshift-service-ca" pod "8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" (UID: "8ed5c5d1-269c-421a-b5b1-765afaf5b9aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:07.944617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.944104 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-metrics-certs" pod "8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" (UID: "8ed5c5d1-269c-421a-b5b1-765afaf5b9aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:01:07.944617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.944367 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-kube-api-access-bbq67" pod "8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" (UID: "8ed5c5d1-269c-421a-b5b1-765afaf5b9aa"). InnerVolumeSpecName "kube-api-access-bbq67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:07.949203 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:07.946030 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-maas-api-tls" pod "8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" (UID: "8ed5c5d1-269c-421a-b5b1-765afaf5b9aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:01:08.007627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:08.006182 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:08.007627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:08.006222 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbq67\" (UniqueName: \"kubernetes.io/projected/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-kube-api-access-bbq67\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:08.007627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:08.006236 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:08.007627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:08.006252 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:08.654654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:08.654624 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" Oct 30 01:01:08.655167 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:08.654611 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt" event={"ID":"8ed5c5d1-269c-421a-b5b1-765afaf5b9aa","Type":"ContainerDied","Data":"62d428bd610023a0e7d313c3d76f72a1598bfc5cafde7ade1ee1a8e0a974e953"} Oct 30 01:01:08.655167 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:08.654786 4927 scope.go:122] "RemoveContainer" containerID="54631015de9dcc7f738a3735878a157cad15c16024d535b5fa365819165ceb0f" Oct 30 01:01:08.737311 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:08.737278 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt"] Oct 30 01:01:08.747267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:08.747237 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601aea-69fb56df9c-6pnmt"] Oct 30 01:01:09.573394 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:09.573359 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" path="/var/lib/kubelet/pods/8ed5c5d1-269c-421a-b5b1-765afaf5b9aa/volumes" Oct 30 01:01:12.894597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:12.865118 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq"] Oct 30 01:01:12.894597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:12.871876 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" podUID="4e207b0f-0bc0-485c-9030-adc9b94a7eac" containerName="maas-api" containerID="cri-o://11b49107e30410eb83c45c2ab42cd5ae35c71ef89cff02154f07da9b275fe5ec" gracePeriod=30 Oct 30 01:01:13.418794 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:13.418749 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" podUID="4e207b0f-0bc0-485c-9030-adc9b94a7eac" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.135:8443/health\": dial tcp 10.132.0.135:8443: connect: connection refused" Oct 30 01:01:14.051653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.031818 4927 generic.go:356] "Generic (PLEG): container finished" podID="4e207b0f-0bc0-485c-9030-adc9b94a7eac" containerID="11b49107e30410eb83c45c2ab42cd5ae35c71ef89cff02154f07da9b275fe5ec" exitCode=0 Oct 30 01:01:14.051653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.032066 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" event={"ID":"4e207b0f-0bc0-485c-9030-adc9b94a7eac","Type":"ContainerDied","Data":"11b49107e30410eb83c45c2ab42cd5ae35c71ef89cff02154f07da9b275fe5ec"} Oct 30 01:01:14.436897 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.436866 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 01:01:14.551321 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.551282 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e207b0f-0bc0-485c-9030-adc9b94a7eac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e207b0f-0bc0-485c-9030-adc9b94a7eac-openshift-service-ca\") pod \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " Oct 30 01:01:14.568682 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.551986 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e207b0f-0bc0-485c-9030-adc9b94a7eac-openshift-service-ca" pod "4e207b0f-0bc0-485c-9030-adc9b94a7eac" (UID: "4e207b0f-0bc0-485c-9030-adc9b94a7eac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:14.585825 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.577933 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-metrics-certs" pod "4e207b0f-0bc0-485c-9030-adc9b94a7eac" (UID: "4e207b0f-0bc0-485c-9030-adc9b94a7eac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:01:14.586107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.569255 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-metrics-certs\") pod \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " Oct 30 01:01:14.586273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.586250 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e207b0f-0bc0-485c-9030-adc9b94a7eac-kube-api-access-drnmg\" (UniqueName: \"kubernetes.io/projected/4e207b0f-0bc0-485c-9030-adc9b94a7eac-kube-api-access-drnmg\") pod \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " Oct 30 01:01:14.586484 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.586465 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-maas-api-tls\") pod \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\" (UID: \"4e207b0f-0bc0-485c-9030-adc9b94a7eac\") " Oct 30 01:01:14.587736 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.587708 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e207b0f-0bc0-485c-9030-adc9b94a7eac-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:14.591714 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.591691 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:14.603400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.603084 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-maas-api-tls" pod "4e207b0f-0bc0-485c-9030-adc9b94a7eac" (UID: "4e207b0f-0bc0-485c-9030-adc9b94a7eac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:01:14.605980 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.605955 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e207b0f-0bc0-485c-9030-adc9b94a7eac-kube-api-access-drnmg" pod "4e207b0f-0bc0-485c-9030-adc9b94a7eac" (UID: "4e207b0f-0bc0-485c-9030-adc9b94a7eac"). InnerVolumeSpecName "kube-api-access-drnmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:14.694719 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.694615 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drnmg\" (UniqueName: \"kubernetes.io/projected/4e207b0f-0bc0-485c-9030-adc9b94a7eac-kube-api-access-drnmg\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:14.694719 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:14.694653 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e207b0f-0bc0-485c-9030-adc9b94a7eac-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:15.043043 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:15.042964 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" Oct 30 01:01:15.043043 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:15.042983 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq" event={"ID":"4e207b0f-0bc0-485c-9030-adc9b94a7eac","Type":"ContainerDied","Data":"e5a6fcadc76d61318485f83fc7c339e369fe7c66b265de660787cc17d1e73048"} Oct 30 01:01:15.043256 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:15.043053 4927 scope.go:122] "RemoveContainer" containerID="11b49107e30410eb83c45c2ab42cd5ae35c71ef89cff02154f07da9b275fe5ec" Oct 30 01:01:15.117092 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:15.117060 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq"] Oct 30 01:01:15.125269 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:15.125239 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32412-59fc67746b-bljhq"] Oct 30 01:01:15.575974 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:15.575925 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e207b0f-0bc0-485c-9030-adc9b94a7eac" path="/var/lib/kubelet/pods/4e207b0f-0bc0-485c-9030-adc9b94a7eac/volumes" Oct 30 01:01:16.557807 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.557774 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx"] Oct 30 01:01:16.558732 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.558713 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4e207b0f-0bc0-485c-9030-adc9b94a7eac" containerName="maas-api" Oct 30 01:01:16.558802 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.558735 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e207b0f-0bc0-485c-9030-adc9b94a7eac" containerName="maas-api" Oct 30 01:01:16.558840 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.558809 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" containerName="maas-api" Oct 30 01:01:16.558840 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.558817 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" containerName="maas-api" Oct 30 01:01:16.559042 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.559028 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed5c5d1-269c-421a-b5b1-765afaf5b9aa" containerName="maas-api" Oct 30 01:01:16.559087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.559051 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e207b0f-0bc0-485c-9030-adc9b94a7eac" containerName="maas-api" Oct 30 01:01:16.564677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.564656 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.594975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.594944 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-820c04-serving-cert\"" Oct 30 01:01:16.601382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.601356 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx"] Oct 30 01:01:16.626885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.626854 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-metrics-certs\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.627056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.627017 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.627229 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.627209 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0c2d70-f1da-49bb-8969-3c78aad572b1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.627304 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.627260 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpwlv\" (UniqueName: \"kubernetes.io/projected/1e0c2d70-f1da-49bb-8969-3c78aad572b1-kube-api-access-vpwlv\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.730482 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.730431 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-metrics-certs\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.730888 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.730864 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.731159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.731138 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0c2d70-f1da-49bb-8969-3c78aad572b1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.731284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.731268 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpwlv\" (UniqueName: \"kubernetes.io/projected/1e0c2d70-f1da-49bb-8969-3c78aad572b1-kube-api-access-vpwlv\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.732715 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:01:16.732692 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-serving-cert: secret "maas-api-e2e-inf-a-820c04-serving-cert" not found Oct 30 01:01:16.733743 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:01:16.733726 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls podName:1e0c2d70-f1da-49bb-8969-3c78aad572b1 nodeName:}" failed. No retries permitted until 2026-10-30 01:01:17.233704338 +0000 UTC m=+1615.805630542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls") pod "maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" (UID: "1e0c2d70-f1da-49bb-8969-3c78aad572b1") : secret "maas-api-e2e-inf-a-820c04-serving-cert" not found Oct 30 01:01:16.734193 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.733610 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0c2d70-f1da-49bb-8969-3c78aad572b1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.788435 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.788361 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-metrics-certs\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:16.788765 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:16.788708 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpwlv\" (UniqueName: \"kubernetes.io/projected/1e0c2d70-f1da-49bb-8969-3c78aad572b1-kube-api-access-vpwlv\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:17.241769 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:17.241733 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:17.244278 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:17.244256 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls\") pod \"maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:17.482364 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:17.482321 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:17.823533 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:17.823505 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx"] Oct 30 01:01:18.136564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:18.136340 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" event={"ID":"1e0c2d70-f1da-49bb-8969-3c78aad572b1","Type":"ContainerStarted","Data":"6c2936c8237aae15779040649aead00ec7dbe031ade80cf52c6281b44820c51b"} Oct 30 01:01:19.149096 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.149056 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" event={"ID":"1e0c2d70-f1da-49bb-8969-3c78aad572b1","Type":"ContainerStarted","Data":"0918793ad5038eacd7583891209cc090e0f73ffca75adf42a7689aa8281d59cf"} Oct 30 01:01:19.149508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.149160 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:19.222118 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.222064 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" podStartSLOduration=2.723807635 podStartE2EDuration="3.222050911s" totalImagesPullingTime="498.243276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:19.21875035 +0000 UTC m=+1617.790676571" watchObservedRunningTime="2026-10-30 01:01:19.222050911 +0000 UTC m=+1617.793977170" Oct 30 01:01:19.305464 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.305427 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6"] Oct 30 01:01:19.314917 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.314887 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.337267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.337224 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6"] Oct 30 01:01:19.357316 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.357272 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-49dfaf-openshift-default-dockercfg-5v6p4\"" Oct 30 01:01:19.393259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.391518 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5045c19-6086-4a68-801c-e3f7b3c098a3-istiod-ca-cert\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.393259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.391670 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-data\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.393259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.391832 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-podinfo\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.393259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.391897 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5tf\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-kube-api-access-fg5tf\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.393259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.391928 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-token\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.393259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.391973 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.393259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.392000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-envoy\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.393259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.392033 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-certs\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.393259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.392100 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-credential-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.495876 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-credential-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.496044 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5045c19-6086-4a68-801c-e3f7b3c098a3-istiod-ca-cert\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.496154 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-data\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.496291 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-podinfo\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.496351 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5tf\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-kube-api-access-fg5tf\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.496379 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-token\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.496431 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.496462 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-envoy\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.496497 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-certs\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.496990 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-certs\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.497669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.497242 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-credential-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.513596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.504044 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-data\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.513596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.504252 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.513596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.509502 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5045c19-6086-4a68-801c-e3f7b3c098a3-istiod-ca-cert\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.532604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.529263 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5tf\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-kube-api-access-fg5tf\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.585596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.585537 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-envoy\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.622592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.615941 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-podinfo\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.670639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.645746 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-token\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:19.931635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:19.931595 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:20.272771 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:01:20.272730 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5045c19_6086_4a68_801c_e3f7b3c098a3.slice/crio-ef5ce404eaaf1cd0f84c09552489b708e61e3d556f69b46b873d1a725dba71a1 WatchSource:0}: Error finding container ef5ce404eaaf1cd0f84c09552489b708e61e3d556f69b46b873d1a725dba71a1: Status 404 returned error can't find the container with id ef5ce404eaaf1cd0f84c09552489b708e61e3d556f69b46b873d1a725dba71a1 Oct 30 01:01:20.274368 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:20.274343 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6"] Oct 30 01:01:20.276150 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:20.276117 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:20.276218 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:20.276186 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:20.276251 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:20.276216 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:20.462030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:20.461994 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn"] Oct 30 01:01:20.462280 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:20.462250 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" containerID="cri-o://ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b" gracePeriod=30 Oct 30 01:01:20.720185 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:20.720149 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b"] Oct 30 01:01:20.720388 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:20.720361 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" containerID="cri-o://cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263" gracePeriod=30 Oct 30 01:01:21.170473 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:21.170434 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" event={"ID":"d5045c19-6086-4a68-801c-e3f7b3c098a3","Type":"ContainerStarted","Data":"e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30"} Oct 30 01:01:21.170678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:21.170499 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" event={"ID":"d5045c19-6086-4a68-801c-e3f7b3c098a3","Type":"ContainerStarted","Data":"ef5ce404eaaf1cd0f84c09552489b708e61e3d556f69b46b873d1a725dba71a1"} Oct 30 01:01:21.244881 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:21.244827 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podStartSLOduration=2.244811183 podStartE2EDuration="2.244811183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:21.242994582 +0000 UTC m=+1619.814920804" watchObservedRunningTime="2026-10-30 01:01:21.244811183 +0000 UTC m=+1619.816737404" Oct 30 01:01:21.932261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:21.932214 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:21.938973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:21.938947 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:22.179090 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:22.179059 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:22.180235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:22.180216 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:01:25.163821 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:25.163785 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:01:25.866712 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:25.866677 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 01:01:26.003379 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.002846 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-socket\") pod \"5327358b-05aa-4bce-8ace-8789224ccbdf\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " Oct 30 01:01:26.003379 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.002935 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-certs\") pod \"5327358b-05aa-4bce-8ace-8789224ccbdf\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " Oct 30 01:01:26.003379 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.002982 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-credential-socket\") pod \"5327358b-05aa-4bce-8ace-8789224ccbdf\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " Oct 30 01:01:26.003379 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.003071 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-data\") pod \"5327358b-05aa-4bce-8ace-8789224ccbdf\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " Oct 30 01:01:26.003379 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.003270 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5327358b-05aa-4bce-8ace-8789224ccbdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327358b-05aa-4bce-8ace-8789224ccbdf-istiod-ca-cert\") pod \"5327358b-05aa-4bce-8ace-8789224ccbdf\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " Oct 30 01:01:26.003379 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.003270 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-socket" pod "5327358b-05aa-4bce-8ace-8789224ccbdf" (UID: "5327358b-05aa-4bce-8ace-8789224ccbdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.003379 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.003309 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-kube-api-access-qhq7k\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-kube-api-access-qhq7k\") pod \"5327358b-05aa-4bce-8ace-8789224ccbdf\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " Oct 30 01:01:26.003884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.003395 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-podinfo\") pod \"5327358b-05aa-4bce-8ace-8789224ccbdf\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " Oct 30 01:01:26.003884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.003459 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-envoy\") pod \"5327358b-05aa-4bce-8ace-8789224ccbdf\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " Oct 30 01:01:26.003884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.003544 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-token\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-token\") pod \"5327358b-05aa-4bce-8ace-8789224ccbdf\" (UID: \"5327358b-05aa-4bce-8ace-8789224ccbdf\") " Oct 30 01:01:26.003884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.003613 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-credential-socket" pod "5327358b-05aa-4bce-8ace-8789224ccbdf" (UID: "5327358b-05aa-4bce-8ace-8789224ccbdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.003884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.003796 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-certs" pod "5327358b-05aa-4bce-8ace-8789224ccbdf" (UID: "5327358b-05aa-4bce-8ace-8789224ccbdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.004173 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.004150 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-data" pod "5327358b-05aa-4bce-8ace-8789224ccbdf" (UID: "5327358b-05aa-4bce-8ace-8789224ccbdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.004441 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.004417 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.004441 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.004442 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.004701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.004456 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.004701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.004469 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.005939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.005718 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5327358b-05aa-4bce-8ace-8789224ccbdf-istiod-ca-cert" pod "5327358b-05aa-4bce-8ace-8789224ccbdf" (UID: "5327358b-05aa-4bce-8ace-8789224ccbdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:26.009062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.008999 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-token" pod "5327358b-05aa-4bce-8ace-8789224ccbdf" (UID: "5327358b-05aa-4bce-8ace-8789224ccbdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.009062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.009003 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-podinfo" pod "5327358b-05aa-4bce-8ace-8789224ccbdf" (UID: "5327358b-05aa-4bce-8ace-8789224ccbdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:26.009235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.009199 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-kube-api-access-qhq7k" pod "5327358b-05aa-4bce-8ace-8789224ccbdf" (UID: "5327358b-05aa-4bce-8ace-8789224ccbdf"). InnerVolumeSpecName "kube-api-access-qhq7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.009467 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.009441 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-envoy" pod "5327358b-05aa-4bce-8ace-8789224ccbdf" (UID: "5327358b-05aa-4bce-8ace-8789224ccbdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.049182 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.049143 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 01:01:26.105742 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.105708 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-credential-socket\") pod \"12d75156-f3c7-4133-8825-93118ecc86b4\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " Oct 30 01:01:26.105990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.105807 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d75156-f3c7-4133-8825-93118ecc86b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d75156-f3c7-4133-8825-93118ecc86b4-istiod-ca-cert\") pod \"12d75156-f3c7-4133-8825-93118ecc86b4\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " Oct 30 01:01:26.105990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.105843 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12d75156-f3c7-4133-8825-93118ecc86b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d75156-f3c7-4133-8825-93118ecc86b4-istio-podinfo\") pod \"12d75156-f3c7-4133-8825-93118ecc86b4\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " Oct 30 01:01:26.105990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.105884 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-kube-api-access-wjnkr\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-kube-api-access-wjnkr\") pod \"12d75156-f3c7-4133-8825-93118ecc86b4\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " Oct 30 01:01:26.106236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.106005 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-envoy\") pod \"12d75156-f3c7-4133-8825-93118ecc86b4\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " Oct 30 01:01:26.106236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.106030 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-credential-socket" pod "12d75156-f3c7-4133-8825-93118ecc86b4" (UID: "12d75156-f3c7-4133-8825-93118ecc86b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.106236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.106124 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-data\") pod \"12d75156-f3c7-4133-8825-93118ecc86b4\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " Oct 30 01:01:26.106236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.106163 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-istio-token\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-istio-token\") pod \"12d75156-f3c7-4133-8825-93118ecc86b4\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " Oct 30 01:01:26.106602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.106358 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d75156-f3c7-4133-8825-93118ecc86b4-istiod-ca-cert" pod "12d75156-f3c7-4133-8825-93118ecc86b4" (UID: "12d75156-f3c7-4133-8825-93118ecc86b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:26.106602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.106369 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-certs\") pod \"12d75156-f3c7-4133-8825-93118ecc86b4\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " Oct 30 01:01:26.106602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.106463 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-socket\") pod \"12d75156-f3c7-4133-8825-93118ecc86b4\" (UID: \"12d75156-f3c7-4133-8825-93118ecc86b4\") " Oct 30 01:01:26.106860 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.106836 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-data" pod "12d75156-f3c7-4133-8825-93118ecc86b4" (UID: "12d75156-f3c7-4133-8825-93118ecc86b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.108044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.106920 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-socket" pod "12d75156-f3c7-4133-8825-93118ecc86b4" (UID: "12d75156-f3c7-4133-8825-93118ecc86b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.108044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.107029 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-certs" pod "12d75156-f3c7-4133-8825-93118ecc86b4" (UID: "12d75156-f3c7-4133-8825-93118ecc86b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.108044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.107306 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.108044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.107324 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d75156-f3c7-4133-8825-93118ecc86b4-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.108044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.107338 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327358b-05aa-4bce-8ace-8789224ccbdf-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.108044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.107358 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhq7k\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-kube-api-access-qhq7k\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.108044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.107372 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.108044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.107385 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.108044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.107398 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5327358b-05aa-4bce-8ace-8789224ccbdf-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.108895 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.108867 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-envoy" pod "12d75156-f3c7-4133-8825-93118ecc86b4" (UID: "12d75156-f3c7-4133-8825-93118ecc86b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:26.109008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.108964 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12d75156-f3c7-4133-8825-93118ecc86b4-istio-podinfo" pod "12d75156-f3c7-4133-8825-93118ecc86b4" (UID: "12d75156-f3c7-4133-8825-93118ecc86b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:26.109696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.109670 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-kube-api-access-wjnkr" pod "12d75156-f3c7-4133-8825-93118ecc86b4" (UID: "12d75156-f3c7-4133-8825-93118ecc86b4"). InnerVolumeSpecName "kube-api-access-wjnkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.110015 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.109979 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-istio-token" pod "12d75156-f3c7-4133-8825-93118ecc86b4" (UID: "12d75156-f3c7-4133-8825-93118ecc86b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:26.161377 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.161337 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56"] Oct 30 01:01:26.162759 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.162732 4927 cpu_manager.go:414] "containerMap: removing container" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" Oct 30 01:01:26.162839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.162761 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" Oct 30 01:01:26.162839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.162779 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" Oct 30 01:01:26.162839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.162787 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" Oct 30 01:01:26.163091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.163066 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" containerName="istio-proxy" Oct 30 01:01:26.163091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.163090 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerName="istio-proxy" Oct 30 01:01:26.168426 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.168400 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:26.201398 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.201362 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56"] Oct 30 01:01:26.204265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.204230 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-49dfaf-dockercfg-j9hph\"" Oct 30 01:01:26.206180 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.206155 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-49dfaf\"" Oct 30 01:01:26.214439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.214404 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.214439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.214439 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.214700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.214453 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.214700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.214466 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.214700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.214478 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d75156-f3c7-4133-8825-93118ecc86b4-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.214700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.214489 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d75156-f3c7-4133-8825-93118ecc86b4-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.214700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.214504 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjnkr\" (UniqueName: \"kubernetes.io/projected/12d75156-f3c7-4133-8825-93118ecc86b4-kube-api-access-wjnkr\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:26.223210 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.223176 4927 generic.go:356] "Generic (PLEG): container finished" podID="5327358b-05aa-4bce-8ace-8789224ccbdf" containerID="ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b" exitCode=0 Oct 30 01:01:26.223408 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.223387 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" event={"ID":"5327358b-05aa-4bce-8ace-8789224ccbdf","Type":"ContainerDied","Data":"ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b"} Oct 30 01:01:26.223489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.223448 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" event={"ID":"5327358b-05aa-4bce-8ace-8789224ccbdf","Type":"ContainerDied","Data":"5929f2f47e3730c52fcc628888c4c010a764852f11a4c804f3ca45cc9a1c6038"} Oct 30 01:01:26.223489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.223482 4927 scope.go:122] "RemoveContainer" containerID="ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b" Oct 30 01:01:26.223698 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.223681 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn" Oct 30 01:01:26.228176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.228145 4927 generic.go:356] "Generic (PLEG): container finished" podID="12d75156-f3c7-4133-8825-93118ecc86b4" containerID="cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263" exitCode=0 Oct 30 01:01:26.228302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.228179 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" event={"ID":"12d75156-f3c7-4133-8825-93118ecc86b4","Type":"ContainerDied","Data":"cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263"} Oct 30 01:01:26.228302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.228161 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" Oct 30 01:01:26.228302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.228229 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b" event={"ID":"12d75156-f3c7-4133-8825-93118ecc86b4","Type":"ContainerDied","Data":"c7c7761f6a22b9b1c206dcf5b12868adc40e9640cd8be568f3ea10682b45d78a"} Oct 30 01:01:26.269625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.269116 4927 scope.go:122] "RemoveContainer" containerID="ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b" Oct 30 01:01:26.269726 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:01:26.269663 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b\": container with ID starting with ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b not found: ID does not exist" containerID="ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b" Oct 30 01:01:26.269726 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.269704 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b"} err="failed to get container status \"ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b\": rpc error: code = NotFound desc = could not find container \"ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b\": container with ID starting with ce08db67cdcad6ec038f136145e088d3e931560b61668ceeda35c10abb34480b not found: ID does not exist" Oct 30 01:01:26.269846 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.269729 4927 scope.go:122] "RemoveContainer" containerID="cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263" Oct 30 01:01:26.296701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.296589 4927 scope.go:122] "RemoveContainer" containerID="cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263" Oct 30 01:01:26.298723 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:01:26.298673 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263\": container with ID starting with cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263 not found: ID does not exist" containerID="cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263" Oct 30 01:01:26.298891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.298734 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263"} err="failed to get container status \"cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263\": rpc error: code = NotFound desc = could not find container \"cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263\": container with ID starting with cbaae0f8405eb7c3d9aaca90d958483d2a4ffabb72c6fa26274dd49be9479263 not found: ID does not exist" Oct 30 01:01:26.318959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.318137 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ms5w\" (UniqueName: \"kubernetes.io/projected/3ffc62a3-2f1e-475d-9073-2ecab906fe07-kube-api-access-8ms5w\") pod \"payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56\" (UID: \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:26.318959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.318300 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffc62a3-2f1e-475d-9073-2ecab906fe07-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56\" (UID: \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:26.331628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.331589 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn"] Oct 30 01:01:26.337289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.337253 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-lk4pn"] Oct 30 01:01:26.492591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.478881 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffc62a3-2f1e-475d-9073-2ecab906fe07-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56\" (UID: \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:26.492591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.479245 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ms5w\" (UniqueName: \"kubernetes.io/projected/3ffc62a3-2f1e-475d-9073-2ecab906fe07-kube-api-access-8ms5w\") pod \"payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56\" (UID: \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:26.492591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.480840 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffc62a3-2f1e-475d-9073-2ecab906fe07-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56\" (UID: \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:26.554641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.536238 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ms5w\" (UniqueName: \"kubernetes.io/projected/3ffc62a3-2f1e-475d-9073-2ecab906fe07-kube-api-access-8ms5w\") pod \"payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56\" (UID: \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:26.589596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.573630 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b"] Oct 30 01:01:26.593597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.592635 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-xh96b"] Oct 30 01:01:26.797606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:26.790506 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:27.608067 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:27.607964 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d75156-f3c7-4133-8825-93118ecc86b4" path="/var/lib/kubelet/pods/12d75156-f3c7-4133-8825-93118ecc86b4/volumes" Oct 30 01:01:27.608674 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:27.608658 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5327358b-05aa-4bce-8ace-8789224ccbdf" path="/var/lib/kubelet/pods/5327358b-05aa-4bce-8ace-8789224ccbdf/volumes" Oct 30 01:01:27.610758 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:27.610730 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56"] Oct 30 01:01:27.613714 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:01:27.613684 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffc62a3_2f1e_475d_9073_2ecab906fe07.slice/crio-9b618e8524c46484c7dbce1c7a663dcab69550cbc504d62284cf6e7467f28dcd WatchSource:0}: Error finding container 9b618e8524c46484c7dbce1c7a663dcab69550cbc504d62284cf6e7467f28dcd: Status 404 returned error can't find the container with id 9b618e8524c46484c7dbce1c7a663dcab69550cbc504d62284cf6e7467f28dcd Oct 30 01:01:28.399290 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:28.399248 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" event={"ID":"3ffc62a3-2f1e-475d-9073-2ecab906fe07","Type":"ContainerStarted","Data":"520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3"} Oct 30 01:01:28.399483 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:28.399320 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" event={"ID":"3ffc62a3-2f1e-475d-9073-2ecab906fe07","Type":"ContainerStarted","Data":"9b618e8524c46484c7dbce1c7a663dcab69550cbc504d62284cf6e7467f28dcd"} Oct 30 01:01:28.400617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:28.400547 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:28.445388 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:28.445302 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" podStartSLOduration=2.445288624 podStartE2EDuration="2.445288624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:28.442762755 +0000 UTC m=+1627.014688978" watchObservedRunningTime="2026-10-30 01:01:28.445288624 +0000 UTC m=+1627.017214872" Oct 30 01:01:34.451639 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:01:34.448439 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:01:35.866660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:35.866621 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p"] Oct 30 01:01:35.867143 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:35.866876 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" containerID="cri-o://3a6c89f77701f4e1488ed2699a6ba1a335200c4dfdee67f9c948f70bac51358d" gracePeriod=30 Oct 30 01:01:36.377749 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:36.377708 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq"] Oct 30 01:01:36.377973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:36.377926 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" containerID="cri-o://29189591a7ec02f9603d3899f85d86fd18f55b5a815db83b04d6aa16110b2a0d" gracePeriod=30 Oct 30 01:01:40.418833 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:40.418799 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:01:41.517288 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.517247 4927 generic.go:356] "Generic (PLEG): container finished" podID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerID="29189591a7ec02f9603d3899f85d86fd18f55b5a815db83b04d6aa16110b2a0d" exitCode=0 Oct 30 01:01:41.517837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.517322 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" event={"ID":"005c3d5d-27a6-4464-b973-f7be87174fb6","Type":"ContainerDied","Data":"29189591a7ec02f9603d3899f85d86fd18f55b5a815db83b04d6aa16110b2a0d"} Oct 30 01:01:41.520862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.520836 4927 generic.go:356] "Generic (PLEG): container finished" podID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerID="3a6c89f77701f4e1488ed2699a6ba1a335200c4dfdee67f9c948f70bac51358d" exitCode=0 Oct 30 01:01:41.521005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.520965 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" event={"ID":"b73f1192-fe9b-46bf-91dd-7995e7b270ad","Type":"ContainerDied","Data":"3a6c89f77701f4e1488ed2699a6ba1a335200c4dfdee67f9c948f70bac51358d"} Oct 30 01:01:41.707187 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.700752 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 01:01:41.742357 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.742331 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 01:01:41.837840 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.837808 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-credential-socket\") pod \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " Oct 30 01:01:41.838018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.837918 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-kube-api-access-p2h6c\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-kube-api-access-p2h6c\") pod \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " Oct 30 01:01:41.838018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.837958 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-kube-api-access-qpp4t\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-kube-api-access-qpp4t\") pod \"005c3d5d-27a6-4464-b973-f7be87174fb6\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " Oct 30 01:01:41.838018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.837995 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/005c3d5d-27a6-4464-b973-f7be87174fb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c3d5d-27a6-4464-b973-f7be87174fb6-istiod-ca-cert\") pod \"005c3d5d-27a6-4464-b973-f7be87174fb6\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " Oct 30 01:01:41.838197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838026 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-socket\") pod \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " Oct 30 01:01:41.838197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838058 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-podinfo\") pod \"005c3d5d-27a6-4464-b973-f7be87174fb6\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " Oct 30 01:01:41.838197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838106 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-envoy\") pod \"005c3d5d-27a6-4464-b973-f7be87174fb6\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " Oct 30 01:01:41.838197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838131 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-credential-socket" pod "b73f1192-fe9b-46bf-91dd-7995e7b270ad" (UID: "b73f1192-fe9b-46bf-91dd-7995e7b270ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.838197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838144 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-envoy\") pod \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " Oct 30 01:01:41.838197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838184 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-certs\") pod \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " Oct 30 01:01:41.838595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838227 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-data\") pod \"005c3d5d-27a6-4464-b973-f7be87174fb6\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " Oct 30 01:01:41.838595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838356 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-credential-socket\") pod \"005c3d5d-27a6-4464-b973-f7be87174fb6\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " Oct 30 01:01:41.838595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838394 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-certs\") pod \"005c3d5d-27a6-4464-b973-f7be87174fb6\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " Oct 30 01:01:41.838595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838410 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005c3d5d-27a6-4464-b973-f7be87174fb6-istiod-ca-cert" pod "005c3d5d-27a6-4464-b973-f7be87174fb6" (UID: "005c3d5d-27a6-4464-b973-f7be87174fb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:41.838595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838477 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-token\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-token\") pod \"005c3d5d-27a6-4464-b973-f7be87174fb6\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " Oct 30 01:01:41.838595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838541 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-data\") pod \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " Oct 30 01:01:41.838595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838595 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-socket\") pod \"005c3d5d-27a6-4464-b973-f7be87174fb6\" (UID: \"005c3d5d-27a6-4464-b973-f7be87174fb6\") " Oct 30 01:01:41.838948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838634 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-token\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-token\") pod \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " Oct 30 01:01:41.838948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838698 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istiod-ca-cert\") pod \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " Oct 30 01:01:41.838948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838766 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-podinfo\") pod \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\" (UID: \"b73f1192-fe9b-46bf-91dd-7995e7b270ad\") " Oct 30 01:01:41.838948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.838794 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-socket" pod "b73f1192-fe9b-46bf-91dd-7995e7b270ad" (UID: "b73f1192-fe9b-46bf-91dd-7995e7b270ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.839160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.839008 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-certs" pod "b73f1192-fe9b-46bf-91dd-7995e7b270ad" (UID: "b73f1192-fe9b-46bf-91dd-7995e7b270ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.839217 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.839175 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-data" pod "b73f1192-fe9b-46bf-91dd-7995e7b270ad" (UID: "b73f1192-fe9b-46bf-91dd-7995e7b270ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.839766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.839659 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.839766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.839689 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.839766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.839707 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c3d5d-27a6-4464-b973-f7be87174fb6-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.839766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.839721 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.840176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.839819 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-data" pod "005c3d5d-27a6-4464-b973-f7be87174fb6" (UID: "005c3d5d-27a6-4464-b973-f7be87174fb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.840457 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.839735 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.840839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.840654 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-credential-socket" pod "005c3d5d-27a6-4464-b973-f7be87174fb6" (UID: "005c3d5d-27a6-4464-b973-f7be87174fb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.841041 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.841010 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-certs" pod "005c3d5d-27a6-4464-b973-f7be87174fb6" (UID: "005c3d5d-27a6-4464-b973-f7be87174fb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.841824 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.841411 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-socket" pod "005c3d5d-27a6-4464-b973-f7be87174fb6" (UID: "005c3d5d-27a6-4464-b973-f7be87174fb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.841824 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.841614 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-podinfo" pod "b73f1192-fe9b-46bf-91dd-7995e7b270ad" (UID: "b73f1192-fe9b-46bf-91dd-7995e7b270ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:41.841824 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.841747 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-envoy" pod "b73f1192-fe9b-46bf-91dd-7995e7b270ad" (UID: "b73f1192-fe9b-46bf-91dd-7995e7b270ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.841824 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.841775 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istiod-ca-cert" pod "b73f1192-fe9b-46bf-91dd-7995e7b270ad" (UID: "b73f1192-fe9b-46bf-91dd-7995e7b270ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:41.841824 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.841810 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-kube-api-access-p2h6c" pod "b73f1192-fe9b-46bf-91dd-7995e7b270ad" (UID: "b73f1192-fe9b-46bf-91dd-7995e7b270ad"). InnerVolumeSpecName "kube-api-access-p2h6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:41.842193 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.841844 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-kube-api-access-qpp4t" pod "005c3d5d-27a6-4464-b973-f7be87174fb6" (UID: "005c3d5d-27a6-4464-b973-f7be87174fb6"). InnerVolumeSpecName "kube-api-access-qpp4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:41.842254 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.842223 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-token" pod "b73f1192-fe9b-46bf-91dd-7995e7b270ad" (UID: "b73f1192-fe9b-46bf-91dd-7995e7b270ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:41.842714 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.842685 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-podinfo" pod "005c3d5d-27a6-4464-b973-f7be87174fb6" (UID: "005c3d5d-27a6-4464-b973-f7be87174fb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:01:41.843158 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.843136 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-envoy" pod "005c3d5d-27a6-4464-b973-f7be87174fb6" (UID: "005c3d5d-27a6-4464-b973-f7be87174fb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:01:41.843334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.843315 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-token" pod "005c3d5d-27a6-4464-b973-f7be87174fb6" (UID: "005c3d5d-27a6-4464-b973-f7be87174fb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:41.941863 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941830 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.941863 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941860 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941877 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941890 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941902 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2h6c\" (UniqueName: \"kubernetes.io/projected/b73f1192-fe9b-46bf-91dd-7995e7b270ad-kube-api-access-p2h6c\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941914 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpp4t\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-kube-api-access-qpp4t\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941927 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941940 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941952 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73f1192-fe9b-46bf-91dd-7995e7b270ad-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941963 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941975 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.941992 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c3d5d-27a6-4464-b973-f7be87174fb6-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:41.942084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:41.942007 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005c3d5d-27a6-4464-b973-f7be87174fb6-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:01:42.532197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.532164 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" Oct 30 01:01:42.532197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.532175 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq" event={"ID":"005c3d5d-27a6-4464-b973-f7be87174fb6","Type":"ContainerDied","Data":"f4d98e63fae6d47c172c86eb24a6a49aef0b0d19d96d67c3bf20b28a46b3ef71"} Oct 30 01:01:42.532891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.532245 4927 scope.go:122] "RemoveContainer" containerID="29189591a7ec02f9603d3899f85d86fd18f55b5a815db83b04d6aa16110b2a0d" Oct 30 01:01:42.533864 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.533840 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" Oct 30 01:01:42.534040 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.533854 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p" event={"ID":"b73f1192-fe9b-46bf-91dd-7995e7b270ad","Type":"ContainerDied","Data":"d0f6b066fed11fe0932f4a2a98267f1113f9679e21e265fae67c07b828a2e810"} Oct 30 01:01:42.544382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.544048 4927 scope.go:122] "RemoveContainer" containerID="3a6c89f77701f4e1488ed2699a6ba1a335200c4dfdee67f9c948f70bac51358d" Oct 30 01:01:42.597191 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.597167 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq"] Oct 30 01:01:42.607658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.607633 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-hwhbq"] Oct 30 01:01:42.634698 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.634675 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p"] Oct 30 01:01:42.639216 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:42.639196 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2zx9p"] Oct 30 01:01:43.574222 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:43.574190 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" path="/var/lib/kubelet/pods/005c3d5d-27a6-4464-b973-f7be87174fb6/volumes" Oct 30 01:01:43.574858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:43.574841 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" path="/var/lib/kubelet/pods/b73f1192-fe9b-46bf-91dd-7995e7b270ad/volumes" Oct 30 01:01:45.847871 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.847835 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj"] Oct 30 01:01:45.848918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.848892 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" Oct 30 01:01:45.848918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.848915 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" Oct 30 01:01:45.849134 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.848958 4927 cpu_manager.go:414] "containerMap: removing container" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" Oct 30 01:01:45.849134 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.848966 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" Oct 30 01:01:45.849240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.849160 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73f1192-fe9b-46bf-91dd-7995e7b270ad" containerName="istio-proxy" Oct 30 01:01:45.849240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.849183 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="005c3d5d-27a6-4464-b973-f7be87174fb6" containerName="istio-proxy" Oct 30 01:01:45.855513 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.855489 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:45.874096 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.874069 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1da7b2-openshift-default-dockercfg-zwj8q\"" Oct 30 01:01:45.889825 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:45.889792 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj"] Oct 30 01:01:46.018606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.016727 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-credential-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.018606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.016776 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-certs\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.018606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.016816 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-token\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.018606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.017083 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709ce488-5c45-4694-a7f7-bb177fa1acfe-istiod-ca-cert\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.018606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.017122 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-podinfo\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.018606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.017187 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.018606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.017260 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-envoy\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.018606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.017306 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-data\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.018606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.017344 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqbw\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-kube-api-access-chqbw\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.118842 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-data\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.118933 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqbw\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-kube-api-access-chqbw\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.119183 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-credential-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.119214 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-certs\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.119268 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-token\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.119599 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709ce488-5c45-4694-a7f7-bb177fa1acfe-istiod-ca-cert\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.119629 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-podinfo\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.119677 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.119738 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-envoy\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.121214 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-certs\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.123640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.121515 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-data\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.129400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.129372 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-credential-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.130374 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.130340 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709ce488-5c45-4694-a7f7-bb177fa1acfe-istiod-ca-cert\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.133602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.130820 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.153592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.136704 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-podinfo\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.154117 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.139956 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-envoy\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.180595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.177167 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-token\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.180595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.178239 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqbw\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-kube-api-access-chqbw\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.187948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.187919 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:46.635998 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.635971 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj"] Oct 30 01:01:46.636669 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:01:46.636640 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709ce488_5c45_4694_a7f7_bb177fa1acfe.slice/crio-37836a8563c85efca07cf90f543ba1f41348e1daa0e38d2b0a5dc99a44b284ff WatchSource:0}: Error finding container 37836a8563c85efca07cf90f543ba1f41348e1daa0e38d2b0a5dc99a44b284ff: Status 404 returned error can't find the container with id 37836a8563c85efca07cf90f543ba1f41348e1daa0e38d2b0a5dc99a44b284ff Oct 30 01:01:46.639448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.639408 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:46.639549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.639495 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:46.639549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:46.639540 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:47.583135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:47.583101 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" event={"ID":"709ce488-5c45-4694-a7f7-bb177fa1acfe","Type":"ContainerStarted","Data":"1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b"} Oct 30 01:01:47.583632 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:47.583152 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" event={"ID":"709ce488-5c45-4694-a7f7-bb177fa1acfe","Type":"ContainerStarted","Data":"37836a8563c85efca07cf90f543ba1f41348e1daa0e38d2b0a5dc99a44b284ff"} Oct 30 01:01:47.651669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:47.651608 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podStartSLOduration=2.651590025 podStartE2EDuration="2.651590025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:47.637642732 +0000 UTC m=+1646.209568953" watchObservedRunningTime="2026-10-30 01:01:47.651590025 +0000 UTC m=+1646.223516245" Oct 30 01:01:48.189490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:48.189453 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:48.195718 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:48.195693 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:48.591333 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:48.591260 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:48.592247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:48.592229 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:01:52.914673 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:52.914630 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd"] Oct 30 01:01:53.054764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.046784 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:01:53.100598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.089785 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1da7b2-dockercfg-n9qh7\"" Oct 30 01:01:53.126717 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.126677 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd"] Oct 30 01:01:53.128665 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.128636 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1da7b2\"" Oct 30 01:01:53.207756 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.207677 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf4kx\" (UniqueName: \"kubernetes.io/projected/facaba60-f652-49ae-8371-8f69798f1c53-kube-api-access-bf4kx\") pod \"payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd\" (UID: \"facaba60-f652-49ae-8371-8f69798f1c53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:01:53.207928 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.207832 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facaba60-f652-49ae-8371-8f69798f1c53-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd\" (UID: \"facaba60-f652-49ae-8371-8f69798f1c53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:01:53.309343 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.309296 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf4kx\" (UniqueName: \"kubernetes.io/projected/facaba60-f652-49ae-8371-8f69798f1c53-kube-api-access-bf4kx\") pod \"payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd\" (UID: \"facaba60-f652-49ae-8371-8f69798f1c53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:01:53.309554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.309399 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facaba60-f652-49ae-8371-8f69798f1c53-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd\" (UID: \"facaba60-f652-49ae-8371-8f69798f1c53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:01:53.310212 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.310188 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facaba60-f652-49ae-8371-8f69798f1c53-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd\" (UID: \"facaba60-f652-49ae-8371-8f69798f1c53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:01:53.332256 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.332222 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf4kx\" (UniqueName: \"kubernetes.io/projected/facaba60-f652-49ae-8371-8f69798f1c53-kube-api-access-bf4kx\") pod \"payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd\" (UID: \"facaba60-f652-49ae-8371-8f69798f1c53\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:01:53.446858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:53.446825 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:01:54.158592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:54.152542 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd"] Oct 30 01:01:55.189167 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:55.189119 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" event={"ID":"facaba60-f652-49ae-8371-8f69798f1c53","Type":"ContainerStarted","Data":"a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57"} Oct 30 01:01:55.189626 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:55.189207 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:01:55.189626 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:55.189232 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" event={"ID":"facaba60-f652-49ae-8371-8f69798f1c53","Type":"ContainerStarted","Data":"3a0cfc3d9eb79f2f579034bb8398c8b704a6ac0ebcb685dd86d4ba80b7dffa98"} Oct 30 01:01:55.233512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:01:55.233458 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" podStartSLOduration=3.233445338 podStartE2EDuration="3.233445338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:55.230737954 +0000 UTC m=+1653.802664186" watchObservedRunningTime="2026-10-30 01:01:55.233445338 +0000 UTC m=+1653.805371559" Oct 30 01:02:04.444914 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:02:04.444836 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:02:06.204513 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:06.204476 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:02:06.902867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:06.902801 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv"] Oct 30 01:02:06.928442 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:06.927399 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv"] Oct 30 01:02:06.928442 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:06.927526 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.019695 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.019636 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-49dfaf-serving-cert\"" Oct 30 01:02:07.059802 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.059762 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjbq\" (UniqueName: \"kubernetes.io/projected/92fe5a50-15bf-4ff7-ab5b-9540ce501400-kube-api-access-drjbq\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.059985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.059947 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe5a50-15bf-4ff7-ab5b-9540ce501400-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.060053 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.060003 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-metrics-certs\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.060260 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.060242 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-maas-api-tls\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.161955 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.161867 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-maas-api-tls\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.162147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.161964 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjbq\" (UniqueName: \"kubernetes.io/projected/92fe5a50-15bf-4ff7-ab5b-9540ce501400-kube-api-access-drjbq\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.162147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.162035 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe5a50-15bf-4ff7-ab5b-9540ce501400-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.162147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.162074 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-metrics-certs\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.162775 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.162749 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe5a50-15bf-4ff7-ab5b-9540ce501400-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.164478 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.164455 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-metrics-certs\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.164627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.164606 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-maas-api-tls\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.184724 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.184693 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjbq\" (UniqueName: \"kubernetes.io/projected/92fe5a50-15bf-4ff7-ab5b-9540ce501400-kube-api-access-drjbq\") pod \"maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.323629 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.321513 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:07.682432 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:02:07.682395 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fe5a50_15bf_4ff7_ab5b_9540ce501400.slice/crio-7e1b1719c984ba4a02c5b53236151863b6d20c02bab52cec88fcf87abca49766 WatchSource:0}: Error finding container 7e1b1719c984ba4a02c5b53236151863b6d20c02bab52cec88fcf87abca49766: Status 404 returned error can't find the container with id 7e1b1719c984ba4a02c5b53236151863b6d20c02bab52cec88fcf87abca49766 Oct 30 01:02:07.682624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:07.682596 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv"] Oct 30 01:02:08.562597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:08.562527 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" event={"ID":"92fe5a50-15bf-4ff7-ab5b-9540ce501400","Type":"ContainerStarted","Data":"45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d"} Oct 30 01:02:08.563224 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:08.562609 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" event={"ID":"92fe5a50-15bf-4ff7-ab5b-9540ce501400","Type":"ContainerStarted","Data":"7e1b1719c984ba4a02c5b53236151863b6d20c02bab52cec88fcf87abca49766"} Oct 30 01:02:08.573698 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:08.563811 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:08.610850 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:08.610789 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" podStartSLOduration=2.250365019 podStartE2EDuration="2.610771231s" totalImagesPullingTime="360.406212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:08.604981157 +0000 UTC m=+1667.176907381" watchObservedRunningTime="2026-10-30 01:02:08.610771231 +0000 UTC m=+1667.182697454" Oct 30 01:02:08.855648 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:08.854625 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr"] Oct 30 01:02:09.109187 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.109106 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr"] Oct 30 01:02:09.109536 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.109513 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.134285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.126296 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dd257f-openshift-default-dockercfg-cs8k9\"" Oct 30 01:02:09.284594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.283183 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istiod-ca-cert\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.284594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.283257 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.284594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.283283 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-data\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.284594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.283539 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-credential-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.284594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.283584 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjpk\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-kube-api-access-vwjpk\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.284594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.283617 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-envoy\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.284594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.283679 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-podinfo\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.284594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.283793 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-certs\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.284594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.283840 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-token\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385321 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385296 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-certs\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385472 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385342 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-token\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385472 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385423 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istiod-ca-cert\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385559 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385474 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385559 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385500 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-data\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385735 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-certs\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385773 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-credential-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385798 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjpk\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-kube-api-access-vwjpk\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-envoy\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385861 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.385972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385877 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-podinfo\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.386052 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.385974 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-data\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.386098 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.386064 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-credential-socket\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.386184 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.386165 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istiod-ca-cert\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.388182 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.388155 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-envoy\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.388329 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.388311 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-podinfo\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.408312 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.408290 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-token\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.408625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.408599 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjpk\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-kube-api-access-vwjpk\") pod \"e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.657528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.657450 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:09.987110 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:02:09.987082 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2e24bf_32fb_44c8_a50c_d0b2abacdc5d.slice/crio-2707fe73a1d0da4d5813a47b5c6d723b883fdaecbdc6ed1f35ae666bb32590d6 WatchSource:0}: Error finding container 2707fe73a1d0da4d5813a47b5c6d723b883fdaecbdc6ed1f35ae666bb32590d6: Status 404 returned error can't find the container with id 2707fe73a1d0da4d5813a47b5c6d723b883fdaecbdc6ed1f35ae666bb32590d6 Oct 30 01:02:09.988416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.988385 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr"] Oct 30 01:02:09.990543 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.990509 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:09.990643 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.990594 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:09.990737 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:09.990644 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:10.581861 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:10.581822 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" event={"ID":"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d","Type":"ContainerStarted","Data":"f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6"} Oct 30 01:02:10.582065 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:10.581880 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" event={"ID":"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d","Type":"ContainerStarted","Data":"2707fe73a1d0da4d5813a47b5c6d723b883fdaecbdc6ed1f35ae666bb32590d6"} Oct 30 01:02:10.650354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:10.650307 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" podStartSLOduration=2.650294072 podStartE2EDuration="2.650294072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:10.635799738 +0000 UTC m=+1669.207725960" watchObservedRunningTime="2026-10-30 01:02:10.650294072 +0000 UTC m=+1669.222220294" Oct 30 01:02:10.658211 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:10.658184 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:11.665311 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:11.665281 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:12.598500 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:12.598466 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:12.599597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:12.599547 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:02:15.588852 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:15.588822 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:02:34.444852 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:02:34.444814 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:02:42.242400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.242360 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg"] Oct 30 01:02:42.249298 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.249274 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.265803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.265777 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg"] Oct 30 01:02:42.271489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.271465 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a07920\"/\"openshift-service-ca.crt\"" Oct 30 01:02:42.273184 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.273162 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a07920\"/\"kube-root-ca.crt\"" Oct 30 01:02:42.274652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.274630 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a07920\"/\"test-model-a07920-kserve-self-signed-certs\"" Oct 30 01:02:42.274896 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.274878 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a07920\"/\"default-dockercfg-ff9vh\"" Oct 30 01:02:42.399609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.374695 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-home\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.400053 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.400017 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jhm\" (UniqueName: \"kubernetes.io/projected/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-kube-api-access-45jhm\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.400246 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.400229 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-dshm\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.400393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.400375 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tmp-dir\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.400677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.400653 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tls-certs\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.400838 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.400822 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-model-cache\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.513153 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.513060 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-dshm\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.513604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.513562 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tmp-dir\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.513934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.513901 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tls-certs\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.514524 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.514501 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-model-cache\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.514849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.514829 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-home\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.515101 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.515083 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jhm\" (UniqueName: \"kubernetes.io/projected/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-kube-api-access-45jhm\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.515994 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.515967 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-model-cache\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.516325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.516305 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tmp-dir\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.516962 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.516941 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-home\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.538405 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.538373 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tls-certs\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.538887 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.538860 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-dshm\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.560283 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.560248 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jhm\" (UniqueName: \"kubernetes.io/projected/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-kube-api-access-45jhm\") pod \"test-model-a07920-kserve-67f9d84db5-29chg\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:42.567604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:42.565190 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:43.206160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:43.206131 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg"] Oct 30 01:02:43.206507 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:02:43.206484 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21dc8a75_4b01_4f9c_af4a_4be8298c0b57.slice/crio-9f54e9c40415b1a0e70c00b311924de9101a0986449c46632f3f589a0c0ec3bd WatchSource:0}: Error finding container 9f54e9c40415b1a0e70c00b311924de9101a0986449c46632f3f589a0c0ec3bd: Status 404 returned error can't find the container with id 9f54e9c40415b1a0e70c00b311924de9101a0986449c46632f3f589a0c0ec3bd Oct 30 01:02:43.990981 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:43.990937 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" event={"ID":"21dc8a75-4b01-4f9c-af4a-4be8298c0b57","Type":"ContainerStarted","Data":"cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3"} Oct 30 01:02:43.991370 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:43.990993 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" event={"ID":"21dc8a75-4b01-4f9c-af4a-4be8298c0b57","Type":"ContainerStarted","Data":"9f54e9c40415b1a0e70c00b311924de9101a0986449c46632f3f589a0c0ec3bd"} Oct 30 01:02:44.032684 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.032620 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" podStartSLOduration=2.032600761 podStartE2EDuration="2.032600761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:44.031033218 +0000 UTC m=+1702.602959439" watchObservedRunningTime="2026-10-30 01:02:44.032600761 +0000 UTC m=+1702.604526985" Oct 30 01:02:44.460240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.460202 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm"] Oct 30 01:02:44.466586 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.466546 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.490522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.490488 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-43c02d\"/\"openshift-service-ca.crt\"" Oct 30 01:02:44.490522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.490492 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-43c02d\"/\"default-dockercfg-7r7cx\"" Oct 30 01:02:44.490818 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.490632 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-43c02d\"/\"test-model-43c02d-kserve-self-signed-certs\"" Oct 30 01:02:44.492189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.492161 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm"] Oct 30 01:02:44.493686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.492280 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-43c02d\"/\"kube-root-ca.crt\"" Oct 30 01:02:44.566298 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.566266 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:44.588336 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.588298 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-dshm\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.588507 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.588491 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-model-cache\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.588607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.588580 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-home\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.588669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.588632 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tls-certs\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.588868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.588847 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tmp-dir\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.589173 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.589149 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6gc7\" (UniqueName: \"kubernetes.io/projected/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-kube-api-access-p6gc7\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.691102 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.691061 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tmp-dir\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.691307 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.691289 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gc7\" (UniqueName: \"kubernetes.io/projected/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-kube-api-access-p6gc7\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.691372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.691341 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-dshm\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.691485 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.691459 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tmp-dir\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.691601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.691550 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-model-cache\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.691682 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.691647 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-home\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.691738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.691703 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tls-certs\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.691866 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.691846 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-model-cache\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.692042 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.692020 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-home\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.693719 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.693693 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-dshm\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.694549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.694528 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tls-certs\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.713392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.713328 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6gc7\" (UniqueName: \"kubernetes.io/projected/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-kube-api-access-p6gc7\") pod \"test-model-43c02d-kserve-8b557c447-tbmnm\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:44.809933 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:44.802588 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:45.352382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:45.352346 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm"] Oct 30 01:02:45.354097 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:02:45.354067 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeee589ce_f8fd_49b4_b1f7_406a26c8dfe8.slice/crio-f38fd14927125d91d24966505eb8ce26325ce9a053dbf19fa197c7ca8c71df0d WatchSource:0}: Error finding container f38fd14927125d91d24966505eb8ce26325ce9a053dbf19fa197c7ca8c71df0d: Status 404 returned error can't find the container with id f38fd14927125d91d24966505eb8ce26325ce9a053dbf19fa197c7ca8c71df0d Oct 30 01:02:46.147433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:46.147396 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" event={"ID":"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8","Type":"ContainerStarted","Data":"bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258"} Oct 30 01:02:46.147433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:46.147446 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" event={"ID":"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8","Type":"ContainerStarted","Data":"f38fd14927125d91d24966505eb8ce26325ce9a053dbf19fa197c7ca8c71df0d"} Oct 30 01:02:46.192246 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:46.192195 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" podStartSLOduration=2.192180163 podStartE2EDuration="2.192180163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:46.189609537 +0000 UTC m=+1704.761535753" watchObservedRunningTime="2026-10-30 01:02:46.192180163 +0000 UTC m=+1704.764106387" Oct 30 01:02:46.806087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:46.806053 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:47.553986 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:47.553956 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:02:47.554161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:47.554014 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 01:02:48.179106 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:48.179070 4927 generic.go:356] "Generic (PLEG): container finished" podID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerID="53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83" exitCode=0 Oct 30 01:02:48.179630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:48.179149 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" event={"ID":"060bfdb8-6cf7-403d-85d2-e81900dc442d","Type":"ContainerDied","Data":"53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83"} Oct 30 01:02:48.180013 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:48.179993 4927 scope.go:122] "RemoveContainer" containerID="53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83" Oct 30 01:02:48.181923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:48.181892 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:48.182017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:48.181965 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:48.182017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:48.181996 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:49.190996 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:49.190944 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" event={"ID":"060bfdb8-6cf7-403d-85d2-e81900dc442d","Type":"ContainerStarted","Data":"f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17"} Oct 30 01:02:50.115681 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:50.115642 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:02:50.115919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:50.115900 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:02:50.115967 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:50.115954 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 01:02:51.115823 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:51.115782 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:02:51.116344 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:51.115866 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 01:02:52.115600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:52.115539 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:02:52.115781 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:52.115641 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 01:02:52.565820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:52.565788 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:52.604248 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:52.604214 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:53.115254 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:53.115217 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:02:53.115473 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:53.115284 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 01:02:53.240616 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:53.240590 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:02:54.115901 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:54.115403 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:02:54.115901 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:54.115478 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 01:02:54.831732 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:54.831694 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:54.847242 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:54.847209 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:55.115934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.115843 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:02:55.116305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.115931 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 01:02:55.321258 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.321227 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:02:55.839844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.839808 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh"] Oct 30 01:02:55.845953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.845928 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:55.869825 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.869793 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh"] Oct 30 01:02:55.870283 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.870109 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2a8b26\"/\"kube-root-ca.crt\"" Oct 30 01:02:55.870466 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.870134 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2a8b26\"/\"openshift-service-ca.crt\"" Oct 30 01:02:55.893901 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.893871 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2a8b26\"/\"test-model-2a8b26-kserve-self-signed-certs\"" Oct 30 01:02:55.906937 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:55.904969 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2a8b26\"/\"default-dockercfg-775zd\"" Oct 30 01:02:56.024587 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.024548 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f086a848-42f7-4736-bd76-dfbffd894327-tls-certs\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.024764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.024693 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-home\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.024764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.024742 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-model-cache\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.024838 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.024779 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-tmp-dir\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.024962 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.024936 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-dshm\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.024998 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.024983 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59z9\" (UniqueName: \"kubernetes.io/projected/f086a848-42f7-4736-bd76-dfbffd894327-kube-api-access-n59z9\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.128694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.126903 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-dshm\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.128694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.126969 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59z9\" (UniqueName: \"kubernetes.io/projected/f086a848-42f7-4736-bd76-dfbffd894327-kube-api-access-n59z9\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.128694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.127219 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f086a848-42f7-4736-bd76-dfbffd894327-tls-certs\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.128694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.127313 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-home\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.128694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.127373 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-model-cache\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.128694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.127417 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-tmp-dir\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.128694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.127942 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-tmp-dir\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.148591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.130628 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-home\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.148983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.140113 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-model-cache\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.183594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.175422 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f086a848-42f7-4736-bd76-dfbffd894327-tls-certs\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.183594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.178901 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:02:56.183594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.179938 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:02:56.186325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.186261 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59z9\" (UniqueName: \"kubernetes.io/projected/f086a848-42f7-4736-bd76-dfbffd894327-kube-api-access-n59z9\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.186829 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.186763 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:02:56.191891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.191835 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-dshm\") pod \"test-model-2a8b26-kserve-54867b88f7-x4cxh\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:56.476287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:56.476194 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:57.143506 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:57.142406 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh"] Oct 30 01:02:57.182598 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:02:57.179895 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf086a848_42f7_4736_bd76_dfbffd894327.slice/crio-43d6f18f931f9ab06bd4889b6cde66c923e18cff3b890ff0f178dc35a443019e WatchSource:0}: Error finding container 43d6f18f931f9ab06bd4889b6cde66c923e18cff3b890ff0f178dc35a443019e: Status 404 returned error can't find the container with id 43d6f18f931f9ab06bd4889b6cde66c923e18cff3b890ff0f178dc35a443019e Oct 30 01:02:57.467648 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:57.463921 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" event={"ID":"f086a848-42f7-4736-bd76-dfbffd894327","Type":"ContainerStarted","Data":"43d6f18f931f9ab06bd4889b6cde66c923e18cff3b890ff0f178dc35a443019e"} Oct 30 01:02:58.473479 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:58.473445 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" event={"ID":"f086a848-42f7-4736-bd76-dfbffd894327","Type":"ContainerStarted","Data":"835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44"} Oct 30 01:02:58.498702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:58.498670 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:02:58.525286 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:02:58.525229 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" podStartSLOduration=3.525215575 podStartE2EDuration="3.525215575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:58.520592456 +0000 UTC m=+1717.092518673" watchObservedRunningTime="2026-10-30 01:02:58.525215575 +0000 UTC m=+1717.097141796" Oct 30 01:03:03.261215 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.261174 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s"] Oct 30 01:03:03.269006 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.268973 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:03.288191 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.288157 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s"] Oct 30 01:03:03.289892 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.289863 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-97ceef-dockercfg-9bv2v\"" Oct 30 01:03:03.387007 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.386955 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxb5q\" (UniqueName: \"kubernetes.io/projected/865e10fe-0a6f-40bf-84c4-5f8b0153d891-kube-api-access-rxb5q\") pod \"payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s\" (UID: \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:03.387401 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.387365 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865e10fe-0a6f-40bf-84c4-5f8b0153d891-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s\" (UID: \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:03.406952 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.388991 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-97ceef\"" Oct 30 01:03:03.800809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.800774 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxb5q\" (UniqueName: \"kubernetes.io/projected/865e10fe-0a6f-40bf-84c4-5f8b0153d891-kube-api-access-rxb5q\") pod \"payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s\" (UID: \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:03.801026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.800881 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865e10fe-0a6f-40bf-84c4-5f8b0153d891-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s\" (UID: \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:03.801690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.801665 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865e10fe-0a6f-40bf-84c4-5f8b0153d891-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s\" (UID: \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:03.823498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.823473 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxb5q\" (UniqueName: \"kubernetes.io/projected/865e10fe-0a6f-40bf-84c4-5f8b0153d891-kube-api-access-rxb5q\") pod \"payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s\" (UID: \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:03.905900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:03.905864 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:04.224592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:04.224551 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s"] Oct 30 01:03:04.224999 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:03:04.224961 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865e10fe_0a6f_40bf_84c4_5f8b0153d891.slice/crio-cddbe6833bc58185e7b4fd0f7be29015db5c2a5e1a34d2ee5383196c3b2921f8 WatchSource:0}: Error finding container cddbe6833bc58185e7b4fd0f7be29015db5c2a5e1a34d2ee5383196c3b2921f8: Status 404 returned error can't find the container with id cddbe6833bc58185e7b4fd0f7be29015db5c2a5e1a34d2ee5383196c3b2921f8 Oct 30 01:03:04.593529 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:03:04.593436 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:03:04.764094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:04.764057 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" event={"ID":"865e10fe-0a6f-40bf-84c4-5f8b0153d891","Type":"ContainerStarted","Data":"be7a8243d58c8efd71db81a54684720e34621a68415235675125bb8f650b2d45"} Oct 30 01:03:04.764363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:04.764341 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" event={"ID":"865e10fe-0a6f-40bf-84c4-5f8b0153d891","Type":"ContainerStarted","Data":"cddbe6833bc58185e7b4fd0f7be29015db5c2a5e1a34d2ee5383196c3b2921f8"} Oct 30 01:03:04.765754 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:04.765731 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:04.868696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:04.859009 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" podStartSLOduration=1.8589883820000002 podStartE2EDuration="1.858988382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:04.81629322 +0000 UTC m=+1723.388219453" watchObservedRunningTime="2026-10-30 01:03:04.858988382 +0000 UTC m=+1723.430914605" Oct 30 01:03:06.478819 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:06.478785 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:03:06.492629 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:06.492604 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:03:06.788104 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:06.788010 4927 generic.go:356] "Generic (PLEG): container finished" podID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerID="e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30" exitCode=0 Oct 30 01:03:06.788104 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:06.788085 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" event={"ID":"d5045c19-6086-4a68-801c-e3f7b3c098a3","Type":"ContainerDied","Data":"e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30"} Oct 30 01:03:06.788719 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:06.788685 4927 scope.go:122] "RemoveContainer" containerID="e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30" Oct 30 01:03:06.791048 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:06.791010 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:06.791160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:06.791103 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:06.791160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:06.791146 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:06.807658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:06.807638 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:03:07.192596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:07.189698 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:03:07.800201 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:07.800160 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" event={"ID":"d5045c19-6086-4a68-801c-e3f7b3c098a3","Type":"ContainerStarted","Data":"ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9"} Oct 30 01:03:07.932323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:07.932290 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:03:07.933159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:07.933132 4927 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:03:07.933293 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:07.933196 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 01:03:08.947594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:08.933051 4927 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:03:08.947594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:08.933120 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 01:03:09.932588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:09.932532 4927 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:03:09.932792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:09.932629 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 01:03:10.411858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.411806 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7"] Oct 30 01:03:10.419772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.419740 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.446468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.446442 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f96ede-openshift-default-dockercfg-47dgx\"" Oct 30 01:03:10.447404 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.447378 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7"] Oct 30 01:03:10.485607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.484992 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj94g\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-kube-api-access-pj94g\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.485607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.485071 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-token\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.485607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.485100 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4711fa-e187-441c-8015-0fb71f9cd69f-istiod-ca-cert\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.485607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.485146 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-credential-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.485607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.485182 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-certs\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.485607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.485446 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-data\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.485607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.485507 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.485607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.485593 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-podinfo\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.498063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.485657 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-envoy\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.614593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.613641 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-token\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.614593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.613697 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4711fa-e187-441c-8015-0fb71f9cd69f-istiod-ca-cert\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.614593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.613743 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-credential-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.614593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.613777 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-certs\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.614593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.614063 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-data\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.614593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.614129 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.614593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.614196 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-podinfo\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.614593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.614258 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-envoy\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.614593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.614342 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj94g\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-kube-api-access-pj94g\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.626591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.623901 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.626591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.623966 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4711fa-e187-441c-8015-0fb71f9cd69f-istiod-ca-cert\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.626591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.623971 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-certs\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.626591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.624168 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-data\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.651605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.641862 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-envoy\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.651605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.641994 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-credential-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.659642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.659108 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-podinfo\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.668841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.667148 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-token\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.699594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.696264 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj94g\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-kube-api-access-pj94g\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.745619 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.745313 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:10.962650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.961658 4927 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:03:10.962650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:10.961769 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 01:03:11.420833 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:03:11.420793 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4711fa_e187_441c_8015_0fb71f9cd69f.slice/crio-292a00a67710a3b2df54d8b7b206421425546371155f2095b9a56adc2a7e3d2e WatchSource:0}: Error finding container 292a00a67710a3b2df54d8b7b206421425546371155f2095b9a56adc2a7e3d2e: Status 404 returned error can't find the container with id 292a00a67710a3b2df54d8b7b206421425546371155f2095b9a56adc2a7e3d2e Oct 30 01:03:11.421265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:11.421033 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7"] Oct 30 01:03:11.424225 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:11.424192 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:11.424316 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:11.424260 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:11.424316 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:11.424292 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:11.916677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:11.916644 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" event={"ID":"0e4711fa-e187-441c-8015-0fb71f9cd69f","Type":"ContainerStarted","Data":"4079a1a32d247cd9f68a51b75301b2374dd8c4ee29ceceadd10e32df6fac92f7"} Oct 30 01:03:11.916864 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:11.916699 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" event={"ID":"0e4711fa-e187-441c-8015-0fb71f9cd69f","Type":"ContainerStarted","Data":"292a00a67710a3b2df54d8b7b206421425546371155f2095b9a56adc2a7e3d2e"} Oct 30 01:03:11.932625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:11.932557 4927 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:03:11.932802 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:11.932649 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 01:03:11.965204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:11.965141 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" podStartSLOduration=1.965123615 podStartE2EDuration="1.965123615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:11.959400255 +0000 UTC m=+1730.531326478" watchObservedRunningTime="2026-10-30 01:03:11.965123615 +0000 UTC m=+1730.537049841" Oct 30 01:03:12.747051 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:12.747006 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:12.753983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:12.753957 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:12.966236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:12.966192 4927 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:03:12.966440 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:12.966275 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 01:03:13.000705 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:13.000617 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:13.002876 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:13.002855 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:03:13.932074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:13.932036 4927 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:03:13.932463 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:13.932094 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 01:03:14.937196 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:14.937164 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:03:14.937673 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:14.937656 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:03:14.938376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:14.938359 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:03:16.796038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:16.796005 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:03:34.466454 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:03:34.466367 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:03:35.514877 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.514844 4927 generic.go:356] "Generic (PLEG): container finished" podID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerID="f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6" exitCode=0 Oct 30 01:03:35.515346 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.514915 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" event={"ID":"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d","Type":"ContainerDied","Data":"f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6"} Oct 30 01:03:35.515422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.515407 4927 scope.go:122] "RemoveContainer" containerID="f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6" Oct 30 01:03:35.517490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.517455 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:35.517596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.517540 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:35.517645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.517592 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:35.837122 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.837033 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8"] Oct 30 01:03:35.843940 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.843905 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:35.860169 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.860141 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ae06fe-serving-cert\"" Oct 30 01:03:35.865943 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.865909 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8"] Oct 30 01:03:35.960903 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.960864 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:35.961126 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.961105 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:35.961283 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.961261 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngxv\" (UniqueName: \"kubernetes.io/projected/c0df2fee-b578-4ce6-995a-405e3edeeba3-kube-api-access-jngxv\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:35.961365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:35.961350 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df2fee-b578-4ce6-995a-405e3edeeba3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.063004 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.062955 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df2fee-b578-4ce6-995a-405e3edeeba3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.063234 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.063074 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.063444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.063421 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.063606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.063585 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngxv\" (UniqueName: \"kubernetes.io/projected/c0df2fee-b578-4ce6-995a-405e3edeeba3-kube-api-access-jngxv\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.065598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.064837 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df2fee-b578-4ce6-995a-405e3edeeba3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.068736 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.068704 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.099695 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.092138 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.103550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.103469 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngxv\" (UniqueName: \"kubernetes.io/projected/c0df2fee-b578-4ce6-995a-405e3edeeba3-kube-api-access-jngxv\") pod \"maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.175601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.169588 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:36.513764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.513721 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8"] Oct 30 01:03:36.529029 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.528999 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:03:36.537830 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.537798 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" event={"ID":"c0df2fee-b578-4ce6-995a-405e3edeeba3","Type":"ContainerStarted","Data":"bc26b49dd35dcb8fd3bd77a11c737628bc6a10b86618f534aa12479f606adb70"} Oct 30 01:03:36.540155 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.540132 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" event={"ID":"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d","Type":"ContainerStarted","Data":"9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84"} Oct 30 01:03:36.657716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.657687 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:03:36.657925 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.657907 4927 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:03:36.657973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:36.657949 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 01:03:37.552451 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:37.552409 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" event={"ID":"c0df2fee-b578-4ce6-995a-405e3edeeba3","Type":"ContainerStarted","Data":"2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f"} Oct 30 01:03:37.552853 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:37.552598 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:37.605268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:37.605175 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" podStartSLOduration=2.225585706 podStartE2EDuration="2.605157479s" totalImagesPullingTime="379.571773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:37.594032007 +0000 UTC m=+1756.165958230" watchObservedRunningTime="2026-10-30 01:03:37.605157479 +0000 UTC m=+1756.177083701" Oct 30 01:03:37.658427 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:37.658388 4927 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:03:37.658664 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:37.658461 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 01:03:38.657863 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:38.657822 4927 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:03:38.658316 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:38.657880 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 01:03:39.658904 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:39.658864 4927 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:03:39.659307 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:39.658926 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 01:03:40.658728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:40.658696 4927 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:03:40.658912 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:40.658750 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 01:03:41.713593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:41.711343 4927 patch_prober.go:29] interesting pod/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:03:41.713593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:41.711405 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 01:03:42.598945 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:42.598893 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:03:42.663762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:42.663729 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:03:42.721080 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:42.721053 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:03:43.631225 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:43.631075 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:03:50.379596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.366474 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch"] Oct 30 01:03:50.405128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.405096 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.431882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.428912 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-97ceef-serving-cert\"" Oct 30 01:03:50.467369 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.467316 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpk5r\" (UniqueName: \"kubernetes.io/projected/6a6cc10e-46b7-4122-a64f-ed718babf1b8-kube-api-access-jpk5r\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.475688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.475651 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-metrics-certs\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.521787 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.521753 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-maas-api-tls\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.528596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.522170 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6cc10e-46b7-4122-a64f-ed718babf1b8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.543353 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.528789 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch"] Oct 30 01:03:50.644558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.644461 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-metrics-certs\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.644558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.644536 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-maas-api-tls\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.644869 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.644777 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6cc10e-46b7-4122-a64f-ed718babf1b8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.644929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.644882 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpk5r\" (UniqueName: \"kubernetes.io/projected/6a6cc10e-46b7-4122-a64f-ed718babf1b8-kube-api-access-jpk5r\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.645801 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.645776 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6cc10e-46b7-4122-a64f-ed718babf1b8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.647872 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.647851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-maas-api-tls\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.648120 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.648102 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-metrics-certs\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.711592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.711486 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpk5r\" (UniqueName: \"kubernetes.io/projected/6a6cc10e-46b7-4122-a64f-ed718babf1b8-kube-api-access-jpk5r\") pod \"maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:50.866002 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:50.865969 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:51.186822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:51.186793 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch"] Oct 30 01:03:51.189071 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:03:51.189041 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6cc10e_46b7_4122_a64f_ed718babf1b8.slice/crio-e3e4eb34789ebc2e790d58ddb56d9391251f06d500da3953de1ddc9219188111 WatchSource:0}: Error finding container e3e4eb34789ebc2e790d58ddb56d9391251f06d500da3953de1ddc9219188111: Status 404 returned error can't find the container with id e3e4eb34789ebc2e790d58ddb56d9391251f06d500da3953de1ddc9219188111 Oct 30 01:03:51.836706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:51.836565 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" event={"ID":"6a6cc10e-46b7-4122-a64f-ed718babf1b8","Type":"ContainerStarted","Data":"635f134a15a3bd512c130464a9967b28e46d002889595f05674e49e4104a0b61"} Oct 30 01:03:51.836706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:51.836663 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" event={"ID":"6a6cc10e-46b7-4122-a64f-ed718babf1b8","Type":"ContainerStarted","Data":"e3e4eb34789ebc2e790d58ddb56d9391251f06d500da3953de1ddc9219188111"} Oct 30 01:03:51.836706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:51.836689 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:51.882544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:51.882491 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" podStartSLOduration=1.576489242 podStartE2EDuration="1.882476946s" totalImagesPullingTime="305.987704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:51.877314162 +0000 UTC m=+1770.449240386" watchObservedRunningTime="2026-10-30 01:03:51.882476946 +0000 UTC m=+1770.454403168" Oct 30 01:03:57.852318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:57.852282 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:03:58.457558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.457521 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm"] Oct 30 01:03:58.462950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.462923 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.483249 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.483217 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm"] Oct 30 01:03:58.487849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.487820 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1da7b2\"/\"default-dockercfg-2bg8l\"" Oct 30 01:03:58.488021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.487882 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1da7b2\"/\"test-model-1da7b2-kserve-self-signed-certs\"" Oct 30 01:03:58.488087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.488065 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1da7b2\"/\"kube-root-ca.crt\"" Oct 30 01:03:58.488225 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.488203 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1da7b2\"/\"openshift-service-ca.crt\"" Oct 30 01:03:58.524157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.524120 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-model-cache\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.524366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.524285 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365d784b-a986-49a5-ba10-93a80bdf2630-tls-certs\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.524462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.524395 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-dshm\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.524615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.524542 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzhd\" (UniqueName: \"kubernetes.io/projected/365d784b-a986-49a5-ba10-93a80bdf2630-kube-api-access-hlzhd\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.524615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.524600 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-tmp-dir\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.524768 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.524713 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-home\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.626273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.626236 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzhd\" (UniqueName: \"kubernetes.io/projected/365d784b-a986-49a5-ba10-93a80bdf2630-kube-api-access-hlzhd\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.626273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.626275 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-tmp-dir\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.626545 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.626339 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-home\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.626545 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.626524 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-model-cache\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.626746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.626724 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365d784b-a986-49a5-ba10-93a80bdf2630-tls-certs\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.626746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.626732 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-tmp-dir\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.626872 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.626836 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-home\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.626872 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.626847 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-dshm\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.626961 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.626919 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-model-cache\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.629083 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.629058 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-dshm\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.629204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.629186 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365d784b-a986-49a5-ba10-93a80bdf2630-tls-certs\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.649268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.649240 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzhd\" (UniqueName: \"kubernetes.io/projected/365d784b-a986-49a5-ba10-93a80bdf2630-kube-api-access-hlzhd\") pod \"test-model-1da7b2-kserve-74549b69d9-gjdtm\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:58.789644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:58.786932 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:03:59.254435 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:03:59.254399 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod365d784b_a986_49a5_ba10_93a80bdf2630.slice/crio-2b4e7801bc0ea8944e7043dce9ef56e23466c4436a7abf1f038dac9a1803c3eb WatchSource:0}: Error finding container 2b4e7801bc0ea8944e7043dce9ef56e23466c4436a7abf1f038dac9a1803c3eb: Status 404 returned error can't find the container with id 2b4e7801bc0ea8944e7043dce9ef56e23466c4436a7abf1f038dac9a1803c3eb Oct 30 01:03:59.255813 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:59.255782 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm"] Oct 30 01:03:59.929229 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:59.929191 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" event={"ID":"365d784b-a986-49a5-ba10-93a80bdf2630","Type":"ContainerStarted","Data":"99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef"} Oct 30 01:03:59.929229 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:59.929233 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" event={"ID":"365d784b-a986-49a5-ba10-93a80bdf2630","Type":"ContainerStarted","Data":"2b4e7801bc0ea8944e7043dce9ef56e23466c4436a7abf1f038dac9a1803c3eb"} Oct 30 01:03:59.973977 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:03:59.973918 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" podStartSLOduration=1.973904726 podStartE2EDuration="1.973904726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:59.968678796 +0000 UTC m=+1778.540605032" watchObservedRunningTime="2026-10-30 01:03:59.973904726 +0000 UTC m=+1778.545830948" Oct 30 01:04:00.787423 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:00.787394 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:04:04.445182 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:04:04.445143 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:04:08.794595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:08.790995 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:04:08.829429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:08.829394 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:04:09.068215 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:09.068128 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:04:22.021487 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:22.021455 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:04:22.023041 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:22.023016 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:04:34.444794 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:04:34.444753 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:04:42.980877 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:42.980836 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw"] Oct 30 01:04:42.989077 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:42.989045 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.007829 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.007799 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw"] Oct 30 01:04:43.018949 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.018925 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-969b92\"/\"openshift-service-ca.crt\"" Oct 30 01:04:43.026785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.026759 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-969b92\"/\"kube-root-ca.crt\"" Oct 30 01:04:43.056831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.056800 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-969b92\"/\"default-dockercfg-5vshj\"" Oct 30 01:04:43.057000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.056844 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-969b92\"/\"auth-test-model-969b92-kserve-self-signed-certs\"" Oct 30 01:04:43.065053 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.065024 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-model-cache\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.065218 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.065091 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-dshm\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.065218 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.065167 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ctc6\" (UniqueName: \"kubernetes.io/projected/83d079dd-5919-43c0-b623-f17e7b1bd38d-kube-api-access-8ctc6\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.065421 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.065401 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d079dd-5919-43c0-b623-f17e7b1bd38d-tls-certs\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.065481 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.065449 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-tmp-dir\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.065535 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.065514 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-home\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.166841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.166804 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-model-cache\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.167024 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.166878 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-dshm\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.167024 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.166923 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ctc6\" (UniqueName: \"kubernetes.io/projected/83d079dd-5919-43c0-b623-f17e7b1bd38d-kube-api-access-8ctc6\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.167220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.167201 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d079dd-5919-43c0-b623-f17e7b1bd38d-tls-certs\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.167268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.167213 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-model-cache\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.167333 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.167282 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-tmp-dir\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.167333 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.167325 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-home\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.167653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.167631 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-tmp-dir\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.167722 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.167686 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-home\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.169260 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.169237 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-dshm\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.169709 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.169684 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d079dd-5919-43c0-b623-f17e7b1bd38d-tls-certs\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.189539 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.189510 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ctc6\" (UniqueName: \"kubernetes.io/projected/83d079dd-5919-43c0-b623-f17e7b1bd38d-kube-api-access-8ctc6\") pod \"auth-test-model-969b92-kserve-7b55c9748f-pbhxw\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.320484 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.320166 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:43.907172 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:04:43.907096 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83d079dd_5919_43c0_b623_f17e7b1bd38d.slice/crio-1ef2cf9a45a2a66ee10d652d42c1cef6016bc2c3b7e7e46e0f35f9b656379e43 WatchSource:0}: Error finding container 1ef2cf9a45a2a66ee10d652d42c1cef6016bc2c3b7e7e46e0f35f9b656379e43: Status 404 returned error can't find the container with id 1ef2cf9a45a2a66ee10d652d42c1cef6016bc2c3b7e7e46e0f35f9b656379e43 Oct 30 01:04:43.909137 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:43.909108 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw"] Oct 30 01:04:44.676121 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:44.676077 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" event={"ID":"83d079dd-5919-43c0-b623-f17e7b1bd38d","Type":"ContainerStarted","Data":"c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e"} Oct 30 01:04:44.676713 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:44.676137 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" event={"ID":"83d079dd-5919-43c0-b623-f17e7b1bd38d","Type":"ContainerStarted","Data":"1ef2cf9a45a2a66ee10d652d42c1cef6016bc2c3b7e7e46e0f35f9b656379e43"} Oct 30 01:04:44.721495 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:44.721426 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" podStartSLOduration=2.721413259 podStartE2EDuration="2.721413259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:44.714859832 +0000 UTC m=+1823.286786045" watchObservedRunningTime="2026-10-30 01:04:44.721413259 +0000 UTC m=+1823.293339480" Oct 30 01:04:45.321454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:45.321405 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:53.321719 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:53.321675 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:53.334937 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:53.334910 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:04:53.770748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:04:53.770714 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:05:04.443865 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:05:04.443785 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:05:14.795629 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:14.792706 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" podUID="f086a848-42f7-4736-bd76-dfbffd894327" containerName="main" probeResult="failure" output="Get \"https://10.132.0.153:8000/ready\": context deadline exceeded" Oct 30 01:05:14.973195 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:14.973147 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 01:05:15.065796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:15.065680 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" podUID="365d784b-a986-49a5-ba10-93a80bdf2630" containerName="main" probeResult="failure" output="Get \"https://10.132.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:05:15.249903 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:15.249074 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:05:18.263400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.263350 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m"] Oct 30 01:05:18.270558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.270534 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.271219 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.271018 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m"] Oct 30 01:05:18.295960 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.295931 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49dfaf\"/\"test-model-49dfaf-kserve-self-signed-certs\"" Oct 30 01:05:18.331616 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.324345 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49dfaf\"/\"openshift-service-ca.crt\"" Oct 30 01:05:18.331616 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.327840 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49dfaf\"/\"default-dockercfg-k84t4\"" Oct 30 01:05:18.331616 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.328713 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49dfaf\"/\"kube-root-ca.crt\"" Oct 30 01:05:18.352705 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.352674 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-home\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.352949 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.352923 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tls-certs\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.353257 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.353237 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-model-cache\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.353435 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.353416 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-dshm\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.353618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.353600 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tmp-dir\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.355105 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.355084 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmph\" (UniqueName: \"kubernetes.io/projected/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-kube-api-access-krmph\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.456439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.456399 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tls-certs\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.456649 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.456549 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-model-cache\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.456649 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.456608 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-dshm\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.456795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.456656 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tmp-dir\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.456795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.456698 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmph\" (UniqueName: \"kubernetes.io/projected/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-kube-api-access-krmph\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.456909 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.456820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-home\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.457022 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.456977 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-model-cache\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.457113 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.457046 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tmp-dir\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.457190 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.457169 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-home\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.459167 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.459136 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-dshm\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.459330 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.459314 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tls-certs\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.617408 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.617329 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmph\" (UniqueName: \"kubernetes.io/projected/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-kube-api-access-krmph\") pod \"test-model-49dfaf-kserve-88949d747-p5f6m\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:18.654354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:18.652908 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:19.462010 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:19.461979 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m"] Oct 30 01:05:19.462966 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:05:19.462935 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaa103dd_7f9a_43b8_bbbe_97ca9e80bdc8.slice/crio-f7540737cf2b8e43af36306aa9883ff5b7e61c09bc6527a1b8e432737441849f WatchSource:0}: Error finding container f7540737cf2b8e43af36306aa9883ff5b7e61c09bc6527a1b8e432737441849f: Status 404 returned error can't find the container with id f7540737cf2b8e43af36306aa9883ff5b7e61c09bc6527a1b8e432737441849f Oct 30 01:05:19.633180 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:19.633143 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" event={"ID":"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8","Type":"ContainerStarted","Data":"f7540737cf2b8e43af36306aa9883ff5b7e61c09bc6527a1b8e432737441849f"} Oct 30 01:05:20.644697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:20.644662 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" event={"ID":"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8","Type":"ContainerStarted","Data":"09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f"} Oct 30 01:05:20.655306 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:20.655279 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:20.701887 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:20.701814 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" podStartSLOduration=2.7017933259999998 podStartE2EDuration="2.701793326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:20.695071114 +0000 UTC m=+1859.266997337" watchObservedRunningTime="2026-10-30 01:05:20.701793326 +0000 UTC m=+1859.273719549" Oct 30 01:05:25.711430 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:25.711392 4927 generic.go:356] "Generic (PLEG): container finished" podID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerID="4079a1a32d247cd9f68a51b75301b2374dd8c4ee29ceceadd10e32df6fac92f7" exitCode=0 Oct 30 01:05:25.713893 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:25.711468 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" event={"ID":"0e4711fa-e187-441c-8015-0fb71f9cd69f","Type":"ContainerDied","Data":"4079a1a32d247cd9f68a51b75301b2374dd8c4ee29ceceadd10e32df6fac92f7"} Oct 30 01:05:25.713893 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:25.712091 4927 scope.go:122] "RemoveContainer" containerID="4079a1a32d247cd9f68a51b75301b2374dd8c4ee29ceceadd10e32df6fac92f7" Oct 30 01:05:25.714265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:25.714221 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:05:25.714308 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:25.714292 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:05:25.714341 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:25.714328 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:05:26.739399 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.739361 4927 generic.go:356] "Generic (PLEG): container finished" podID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerID="1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b" exitCode=0 Oct 30 01:05:26.739935 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.739436 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" event={"ID":"709ce488-5c45-4694-a7f7-bb177fa1acfe","Type":"ContainerDied","Data":"1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b"} Oct 30 01:05:26.740123 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.740106 4927 scope.go:122] "RemoveContainer" containerID="1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b" Oct 30 01:05:26.742501 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.742475 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" event={"ID":"0e4711fa-e187-441c-8015-0fb71f9cd69f","Type":"ContainerStarted","Data":"2b0faaec7d116dad7f6568d36ab0dcb19624fd5172cc0b27cfca9bf9794b5cdc"} Oct 30 01:05:26.742627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.742599 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:05:26.742696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.742681 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:05:26.742732 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.742721 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:05:26.747211 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.747189 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:05:26.748767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.748745 4927 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:05:26.748853 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:26.748796 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 01:05:27.123700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.123522 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh"] Oct 30 01:05:27.134307 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.134275 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.140008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.139973 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh"] Oct 30 01:05:27.149604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.149562 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-97ceef\"/\"openshift-service-ca.crt\"" Oct 30 01:05:27.149797 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.149592 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-97ceef\"/\"kube-root-ca.crt\"" Oct 30 01:05:27.149912 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.149672 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-97ceef\"/\"test-model-97ceef-kserve-self-signed-certs\"" Oct 30 01:05:27.150001 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.149731 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-97ceef\"/\"default-dockercfg-rxzhf\"" Oct 30 01:05:27.216906 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.216867 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-dshm\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.217146 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.216931 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a54e8b1-b885-49c6-b803-5cad96e73314-tls-certs\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.217146 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.217132 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-home\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.217287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.217209 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-model-cache\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.217350 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.217313 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-tmp-dir\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.217413 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.217376 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhsf6\" (UniqueName: \"kubernetes.io/projected/6a54e8b1-b885-49c6-b803-5cad96e73314-kube-api-access-vhsf6\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.318949 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.318913 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-home\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.319141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.318979 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-model-cache\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.319141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.319044 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-tmp-dir\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.319141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.319086 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhsf6\" (UniqueName: \"kubernetes.io/projected/6a54e8b1-b885-49c6-b803-5cad96e73314-kube-api-access-vhsf6\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.319261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.319183 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-dshm\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.319261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.319224 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a54e8b1-b885-49c6-b803-5cad96e73314-tls-certs\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.319421 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.319398 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-home\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.319471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.319410 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-model-cache\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.319511 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.319494 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-tmp-dir\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.321899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.321870 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-dshm\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.322139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.322119 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a54e8b1-b885-49c6-b803-5cad96e73314-tls-certs\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.341891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.341858 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhsf6\" (UniqueName: \"kubernetes.io/projected/6a54e8b1-b885-49c6-b803-5cad96e73314-kube-api-access-vhsf6\") pod \"test-model-97ceef-kserve-d4bff645c-prddh\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.461652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.455142 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:27.767638 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.765245 4927 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:05:27.767638 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.765478 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 01:05:27.838497 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:27.837747 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" event={"ID":"709ce488-5c45-4694-a7f7-bb177fa1acfe","Type":"ContainerStarted","Data":"33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc"} Oct 30 01:05:28.009599 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.001699 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:05:28.127656 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.115761 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh"] Oct 30 01:05:28.200212 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.197249 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:05:28.223597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.209718 4927 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:05:28.223597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.209792 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 01:05:28.655799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.655764 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:28.669754 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.669724 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:28.747468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.747428 4927 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:05:28.747654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.747494 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 01:05:28.849799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.849608 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" event={"ID":"6a54e8b1-b885-49c6-b803-5cad96e73314","Type":"ContainerStarted","Data":"732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940"} Oct 30 01:05:28.849799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.849665 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" event={"ID":"6a54e8b1-b885-49c6-b803-5cad96e73314","Type":"ContainerStarted","Data":"a021ba06aec747d95509f15b81faa2b891374b206956e13fe21096e39c588c00"} Oct 30 01:05:28.862080 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.862049 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:05:28.937256 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.937155 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm"] Oct 30 01:05:28.942212 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.942140 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" podStartSLOduration=1.942122335 podStartE2EDuration="1.942122335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:28.897983053 +0000 UTC m=+1867.469909276" watchObservedRunningTime="2026-10-30 01:05:28.942122335 +0000 UTC m=+1867.514048557" Oct 30 01:05:28.947178 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.947149 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:28.955552 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.955516 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm"] Oct 30 01:05:28.963428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.963398 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cf3466\"/\"openshift-service-ca.crt\"" Oct 30 01:05:28.963592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.963523 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cf3466\"/\"kube-root-ca.crt\"" Oct 30 01:05:28.964240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.964217 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cf3466\"/\"test-model-cf3466-kserve-self-signed-certs\"" Oct 30 01:05:28.964389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:28.964371 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cf3466\"/\"default-dockercfg-xjm7f\"" Oct 30 01:05:29.069309 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.067430 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-tmp-dir\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.069309 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.067522 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-home\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.069309 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.067666 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-model-cache\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.069309 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.067722 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-dshm\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.069309 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.067797 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbshg\" (UniqueName: \"kubernetes.io/projected/b99f3492-5f82-45bf-a645-9f28cad2a971-kube-api-access-pbshg\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.069309 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.068036 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99f3492-5f82-45bf-a645-9f28cad2a971-tls-certs\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.170358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.170318 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99f3492-5f82-45bf-a645-9f28cad2a971-tls-certs\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.170858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.170502 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-tmp-dir\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.170858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.170554 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-home\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.170858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.170667 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-model-cache\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.170858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.170711 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-dshm\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.170858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.170774 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbshg\" (UniqueName: \"kubernetes.io/projected/b99f3492-5f82-45bf-a645-9f28cad2a971-kube-api-access-pbshg\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.231487 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.195208 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-home\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.231487 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.195851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-tmp-dir\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.231487 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.196097 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-model-cache\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.232167 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.232105 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-dshm\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.277318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.276293 4927 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:05:29.277318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.276357 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 01:05:29.282540 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.282510 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99f3492-5f82-45bf-a645-9f28cad2a971-tls-certs\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.319860 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.288494 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbshg\" (UniqueName: \"kubernetes.io/projected/b99f3492-5f82-45bf-a645-9f28cad2a971-kube-api-access-pbshg\") pod \"test-model-cf3466-kserve-5c68988b55-6stcm\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.339007 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.338970 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:29.466593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.459674 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:29.746785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.746755 4927 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:05:29.747021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:29.746994 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 01:05:30.231237 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:30.231197 4927 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:05:30.231707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:30.231263 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 01:05:30.233725 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:30.233696 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm"] Oct 30 01:05:30.234303 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:05:30.234275 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb99f3492_5f82_45bf_a645_9f28cad2a971.slice/crio-bd4fa8c3136052ebb94a07a53e42a7b02adf6503c1b6e58c60f1d561666e7683 WatchSource:0}: Error finding container bd4fa8c3136052ebb94a07a53e42a7b02adf6503c1b6e58c60f1d561666e7683: Status 404 returned error can't find the container with id bd4fa8c3136052ebb94a07a53e42a7b02adf6503c1b6e58c60f1d561666e7683 Oct 30 01:05:30.747325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:30.747270 4927 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:05:30.747877 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:30.747345 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 01:05:31.051361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:31.051110 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" event={"ID":"b99f3492-5f82-45bf-a645-9f28cad2a971","Type":"ContainerStarted","Data":"f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692"} Oct 30 01:05:31.051361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:31.051161 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" event={"ID":"b99f3492-5f82-45bf-a645-9f28cad2a971","Type":"ContainerStarted","Data":"bd4fa8c3136052ebb94a07a53e42a7b02adf6503c1b6e58c60f1d561666e7683"} Oct 30 01:05:31.100632 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:31.098971 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" podStartSLOduration=3.098949671 podStartE2EDuration="3.098949671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:31.096015063 +0000 UTC m=+1869.667941287" watchObservedRunningTime="2026-10-30 01:05:31.098949671 +0000 UTC m=+1869.670875894" Oct 30 01:05:31.194000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:31.193962 4927 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:05:31.194296 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:31.194269 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 01:05:31.342646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:31.341671 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:31.747685 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:31.747219 4927 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:05:31.747685 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:31.747291 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 01:05:32.191094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:32.191046 4927 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:05:32.191413 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:32.191377 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 01:05:32.746874 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:32.746839 4927 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:05:32.747284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:32.746934 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 01:05:33.192026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:33.191984 4927 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:05:33.192351 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:33.192323 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 01:05:33.686316 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:33.686279 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:05:33.752345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:33.752311 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:05:33.753611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:33.753561 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:05:34.189252 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:34.189217 4927 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:05:34.189444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:34.189285 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 01:05:34.451608 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:05:34.451491 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:05:35.194697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:35.194657 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:05:35.196067 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:35.196043 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:05:37.456401 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:37.456362 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:37.469631 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:37.469601 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:38.128373 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:38.128344 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:05:39.340645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:39.340607 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:39.353388 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:39.353355 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:40.147785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:40.147753 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:05:50.619500 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.619463 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz"] Oct 30 01:05:50.626039 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.625986 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.650701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.650673 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae06fe\"/\"kube-root-ca.crt\"" Oct 30 01:05:50.650916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.650690 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae06fe\"/\"openshift-service-ca.crt\"" Oct 30 01:05:50.650916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.650785 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae06fe\"/\"auth-test-model-ae06fe-kserve-self-signed-certs\"" Oct 30 01:05:50.651072 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.650860 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz"] Oct 30 01:05:50.725113 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.725078 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae06fe\"/\"default-dockercfg-tfpg9\"" Oct 30 01:05:50.758058 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.758027 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stp2\" (UniqueName: \"kubernetes.io/projected/7ea7d8b4-a022-4240-a420-b2443bca2a03-kube-api-access-8stp2\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.758256 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.758083 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-dshm\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.758256 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.758203 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-tmp-dir\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.758358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.758339 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea7d8b4-a022-4240-a420-b2443bca2a03-tls-certs\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.758406 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.758387 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-model-cache\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.758492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.758476 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-home\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.891645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.877923 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stp2\" (UniqueName: \"kubernetes.io/projected/7ea7d8b4-a022-4240-a420-b2443bca2a03-kube-api-access-8stp2\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.891645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.881708 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-dshm\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.891645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.881817 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-tmp-dir\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.891645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.881988 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea7d8b4-a022-4240-a420-b2443bca2a03-tls-certs\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.891645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.882068 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-model-cache\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.891645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.882171 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-home\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.891645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.882768 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-home\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.899630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.899445 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-model-cache\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.899848 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.899821 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-tmp-dir\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.934608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.922519 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea7d8b4-a022-4240-a420-b2443bca2a03-tls-certs\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:50.963647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:50.957255 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-dshm\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:51.014630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:51.000370 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stp2\" (UniqueName: \"kubernetes.io/projected/7ea7d8b4-a022-4240-a420-b2443bca2a03-kube-api-access-8stp2\") pod \"auth-test-model-ae06fe-kserve-66996467d9-qsgxz\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:51.251226 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:51.251136 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:05:51.590437 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:51.590406 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz"] Oct 30 01:05:52.266760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:52.266720 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" event={"ID":"7ea7d8b4-a022-4240-a420-b2443bca2a03","Type":"ContainerStarted","Data":"c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38"} Oct 30 01:05:52.266760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:52.266766 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" event={"ID":"7ea7d8b4-a022-4240-a420-b2443bca2a03","Type":"ContainerStarted","Data":"db25cec603d45714e5197745a36c480471e07cd8bc27260c3c69ae14bcbc3b97"} Oct 30 01:05:52.324924 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:52.324857 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" podStartSLOduration=2.3248411669999998 podStartE2EDuration="2.324841167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:52.320109025 +0000 UTC m=+1890.892035247" watchObservedRunningTime="2026-10-30 01:05:52.324841167 +0000 UTC m=+1890.896767440" Oct 30 01:05:53.252288 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:05:53.252250 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:06:01.251328 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:01.251287 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:06:01.265870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:01.265838 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:06:01.373301 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:01.373272 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:06:04.449372 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:06:04.449338 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:06:06.432116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:06.432062 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64"] Oct 30 01:06:06.453690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:06.433168 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" containerID="cri-o://de62e63e2098732cf7d2bc7429da8e0342c0c15c7ab3a1cd241c8d43b576c3ac" gracePeriod=30 Oct 30 01:06:06.505608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:06.494543 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p"] Oct 30 01:06:06.505608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:06.494873 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" containerID="cri-o://838f26bc22a2dd80a3c2346ed5671d8d6dab09e8c7f2afa10c13fa9cf16ffe8a" gracePeriod=30 Oct 30 01:06:06.582381 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:06.582339 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8"] Oct 30 01:06:06.582649 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:06.582619 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" containerID="cri-o://1e8f32a8ec256628dcc592493a9680815c534411eab738ba38c4b83aafb55183" gracePeriod=30 Oct 30 01:06:06.601169 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:06.601137 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf"] Oct 30 01:06:06.601395 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:06.601370 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" containerID="cri-o://da4b60830c859647e114963fd70ddef0a29d34ad68902ed8873e9507452a76e9" gracePeriod=30 Oct 30 01:06:11.853440 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:11.811210 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:06:11.853440 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:11.811300 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 01:06:12.763488 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.763451 4927 generic.go:356] "Generic (PLEG): container finished" podID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerID="da4b60830c859647e114963fd70ddef0a29d34ad68902ed8873e9507452a76e9" exitCode=0 Oct 30 01:06:12.763867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.763837 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" event={"ID":"c27bd511-fd29-4697-8cd3-6d9d91ea9228","Type":"ContainerDied","Data":"da4b60830c859647e114963fd70ddef0a29d34ad68902ed8873e9507452a76e9"} Oct 30 01:06:12.768225 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.768198 4927 generic.go:356] "Generic (PLEG): container finished" podID="83a4a673-85e7-412c-bd86-33d62b2be721" containerID="de62e63e2098732cf7d2bc7429da8e0342c0c15c7ab3a1cd241c8d43b576c3ac" exitCode=0 Oct 30 01:06:12.768503 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.768481 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" event={"ID":"83a4a673-85e7-412c-bd86-33d62b2be721","Type":"ContainerDied","Data":"de62e63e2098732cf7d2bc7429da8e0342c0c15c7ab3a1cd241c8d43b576c3ac"} Oct 30 01:06:12.770722 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.770698 4927 generic.go:356] "Generic (PLEG): container finished" podID="218629e5-8db6-4edd-91bc-179727432092" containerID="838f26bc22a2dd80a3c2346ed5671d8d6dab09e8c7f2afa10c13fa9cf16ffe8a" exitCode=0 Oct 30 01:06:12.771434 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.771408 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" event={"ID":"218629e5-8db6-4edd-91bc-179727432092","Type":"ContainerDied","Data":"838f26bc22a2dd80a3c2346ed5671d8d6dab09e8c7f2afa10c13fa9cf16ffe8a"} Oct 30 01:06:12.808809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.808769 4927 generic.go:356] "Generic (PLEG): container finished" podID="01c22e25-5694-4368-9462-5b84f2226eb4" containerID="1e8f32a8ec256628dcc592493a9680815c534411eab738ba38c4b83aafb55183" exitCode=0 Oct 30 01:06:12.808997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.808875 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" event={"ID":"01c22e25-5694-4368-9462-5b84f2226eb4","Type":"ContainerDied","Data":"1e8f32a8ec256628dcc592493a9680815c534411eab738ba38c4b83aafb55183"} Oct 30 01:06:12.812423 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.812387 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 01:06:12.890677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.889811 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 01:06:12.898434 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.898407 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 01:06:12.903072 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.903043 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 01:06:12.908742 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.908715 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-envoy\") pod \"218629e5-8db6-4edd-91bc-179727432092\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " Oct 30 01:06:12.908893 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.908876 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218629e5-8db6-4edd-91bc-179727432092-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218629e5-8db6-4edd-91bc-179727432092-istiod-ca-cert\") pod \"218629e5-8db6-4edd-91bc-179727432092\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " Oct 30 01:06:12.908948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.908932 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-kube-api-access-9rv7q\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-kube-api-access-9rv7q\") pod \"218629e5-8db6-4edd-91bc-179727432092\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " Oct 30 01:06:12.909008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.908985 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-data\") pod \"218629e5-8db6-4edd-91bc-179727432092\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " Oct 30 01:06:12.909063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909052 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-socket\") pod \"218629e5-8db6-4edd-91bc-179727432092\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " Oct 30 01:06:12.909121 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909090 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-certs\") pod \"218629e5-8db6-4edd-91bc-179727432092\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " Oct 30 01:06:12.909174 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909120 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-istio-token\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-istio-token\") pod \"218629e5-8db6-4edd-91bc-179727432092\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " Oct 30 01:06:12.909239 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909184 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-credential-socket\") pod \"218629e5-8db6-4edd-91bc-179727432092\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " Oct 30 01:06:12.909334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909317 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/218629e5-8db6-4edd-91bc-179727432092-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218629e5-8db6-4edd-91bc-179727432092-istio-podinfo\") pod \"218629e5-8db6-4edd-91bc-179727432092\" (UID: \"218629e5-8db6-4edd-91bc-179727432092\") " Oct 30 01:06:12.909393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909344 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-socket" pod "218629e5-8db6-4edd-91bc-179727432092" (UID: "218629e5-8db6-4edd-91bc-179727432092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.909393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909379 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-data" pod "218629e5-8db6-4edd-91bc-179727432092" (UID: "218629e5-8db6-4edd-91bc-179727432092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.909499 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909373 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218629e5-8db6-4edd-91bc-179727432092-istiod-ca-cert" pod "218629e5-8db6-4edd-91bc-179727432092" (UID: "218629e5-8db6-4edd-91bc-179727432092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:12.909499 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909392 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-certs" pod "218629e5-8db6-4edd-91bc-179727432092" (UID: "218629e5-8db6-4edd-91bc-179727432092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.909646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.909621 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-credential-socket" pod "218629e5-8db6-4edd-91bc-179727432092" (UID: "218629e5-8db6-4edd-91bc-179727432092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:12.910085 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.910063 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218629e5-8db6-4edd-91bc-179727432092-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.910085 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.910084 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.910255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.910093 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.910255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.910101 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.910255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.910110 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:12.911738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.911718 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-kube-api-access-9rv7q" pod "218629e5-8db6-4edd-91bc-179727432092" (UID: "218629e5-8db6-4edd-91bc-179727432092"). InnerVolumeSpecName "kube-api-access-9rv7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:12.911965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.911947 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-istio-token" pod "218629e5-8db6-4edd-91bc-179727432092" (UID: "218629e5-8db6-4edd-91bc-179727432092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:12.912013 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.911966 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/218629e5-8db6-4edd-91bc-179727432092-istio-podinfo" pod "218629e5-8db6-4edd-91bc-179727432092" (UID: "218629e5-8db6-4edd-91bc-179727432092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:12.914104 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:12.914085 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-envoy" pod "218629e5-8db6-4edd-91bc-179727432092" (UID: "218629e5-8db6-4edd-91bc-179727432092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.007550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.007467 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm"] Oct 30 01:06:13.011535 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011500 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-socket\") pod \"83a4a673-85e7-412c-bd86-33d62b2be721\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " Oct 30 01:06:13.011706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011616 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-certs\") pod \"01c22e25-5694-4368-9462-5b84f2226eb4\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " Oct 30 01:06:13.011837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011718 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-kube-api-access-q8bjc\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-kube-api-access-q8bjc\") pod \"83a4a673-85e7-412c-bd86-33d62b2be721\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " Oct 30 01:06:13.011837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011753 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-kube-api-access-zjp8z\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-kube-api-access-zjp8z\") pod \"01c22e25-5694-4368-9462-5b84f2226eb4\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " Oct 30 01:06:13.011837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011787 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-certs\") pod \"83a4a673-85e7-412c-bd86-33d62b2be721\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " Oct 30 01:06:13.011837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011794 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-socket" pod "83a4a673-85e7-412c-bd86-33d62b2be721" (UID: "83a4a673-85e7-412c-bd86-33d62b2be721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.011837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011814 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-envoy\") pod \"01c22e25-5694-4368-9462-5b84f2226eb4\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " Oct 30 01:06:13.012139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011888 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-istio-token\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-istio-token\") pod \"01c22e25-5694-4368-9462-5b84f2226eb4\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " Oct 30 01:06:13.012139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011908 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-certs" pod "01c22e25-5694-4368-9462-5b84f2226eb4" (UID: "01c22e25-5694-4368-9462-5b84f2226eb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.012139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011953 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-socket\") pod \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " Oct 30 01:06:13.012139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.011991 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-socket\") pod \"01c22e25-5694-4368-9462-5b84f2226eb4\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " Oct 30 01:06:13.012365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012134 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-certs" pod "83a4a673-85e7-412c-bd86-33d62b2be721" (UID: "83a4a673-85e7-412c-bd86-33d62b2be721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.012365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012218 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-data\") pod \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " Oct 30 01:06:13.012365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012208 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-socket" pod "01c22e25-5694-4368-9462-5b84f2226eb4" (UID: "01c22e25-5694-4368-9462-5b84f2226eb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.012365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012270 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-socket" pod "c27bd511-fd29-4697-8cd3-6d9d91ea9228" (UID: "c27bd511-fd29-4697-8cd3-6d9d91ea9228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.012365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012291 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-data\") pod \"01c22e25-5694-4368-9462-5b84f2226eb4\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " Oct 30 01:06:13.012365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012352 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-token\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-token\") pod \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " Oct 30 01:06:13.012683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012404 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-podinfo\") pod \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " Oct 30 01:06:13.012683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012460 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c22e25-5694-4368-9462-5b84f2226eb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c22e25-5694-4368-9462-5b84f2226eb4-istiod-ca-cert\") pod \"01c22e25-5694-4368-9462-5b84f2226eb4\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " Oct 30 01:06:13.012683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012509 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-data\") pod \"83a4a673-85e7-412c-bd86-33d62b2be721\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " Oct 30 01:06:13.012683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012524 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-data" pod "01c22e25-5694-4368-9462-5b84f2226eb4" (UID: "01c22e25-5694-4368-9462-5b84f2226eb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.012683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012550 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a4a673-85e7-412c-bd86-33d62b2be721-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4a673-85e7-412c-bd86-33d62b2be721-istiod-ca-cert\") pod \"83a4a673-85e7-412c-bd86-33d62b2be721\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " Oct 30 01:06:13.012683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012644 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-istio-token\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-istio-token\") pod \"83a4a673-85e7-412c-bd86-33d62b2be721\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " Oct 30 01:06:13.012683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012682 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-credential-socket\") pod \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012711 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01c22e25-5694-4368-9462-5b84f2226eb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c22e25-5694-4368-9462-5b84f2226eb4-istio-podinfo\") pod \"01c22e25-5694-4368-9462-5b84f2226eb4\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012751 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-envoy\") pod \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012798 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-data" pod "c27bd511-fd29-4697-8cd3-6d9d91ea9228" (UID: "c27bd511-fd29-4697-8cd3-6d9d91ea9228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012835 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-credential-socket\") pod \"01c22e25-5694-4368-9462-5b84f2226eb4\" (UID: \"01c22e25-5694-4368-9462-5b84f2226eb4\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012846 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c22e25-5694-4368-9462-5b84f2226eb4-istiod-ca-cert" pod "01c22e25-5694-4368-9462-5b84f2226eb4" (UID: "01c22e25-5694-4368-9462-5b84f2226eb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.012902 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-credential-socket\") pod \"83a4a673-85e7-412c-bd86-33d62b2be721\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.013151 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-envoy\") pod \"83a4a673-85e7-412c-bd86-33d62b2be721\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.013224 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83a4a673-85e7-412c-bd86-33d62b2be721-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4a673-85e7-412c-bd86-33d62b2be721-istio-podinfo\") pod \"83a4a673-85e7-412c-bd86-33d62b2be721\" (UID: \"83a4a673-85e7-412c-bd86-33d62b2be721\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.013245 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-credential-socket" pod "01c22e25-5694-4368-9462-5b84f2226eb4" (UID: "01c22e25-5694-4368-9462-5b84f2226eb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.013266 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-kube-api-access-d2j48\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-kube-api-access-d2j48\") pod \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.013307 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istiod-ca-cert\") pod \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.013344 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-certs\") pod \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\" (UID: \"c27bd511-fd29-4697-8cd3-6d9d91ea9228\") " Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.013463 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-data" pod "83a4a673-85e7-412c-bd86-33d62b2be721" (UID: "83a4a673-85e7-412c-bd86-33d62b2be721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.013819 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a4a673-85e7-412c-bd86-33d62b2be721-istiod-ca-cert" pod "83a4a673-85e7-412c-bd86-33d62b2be721" (UID: "83a4a673-85e7-412c-bd86-33d62b2be721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014033 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-certs" pod "c27bd511-fd29-4697-8cd3-6d9d91ea9228" (UID: "c27bd511-fd29-4697-8cd3-6d9d91ea9228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.014454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014205 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-credential-socket" pod "83a4a673-85e7-412c-bd86-33d62b2be721" (UID: "83a4a673-85e7-412c-bd86-33d62b2be721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014474 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014492 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014505 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014519 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014530 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014555 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218629e5-8db6-4edd-91bc-179727432092-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014607 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014621 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014637 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014650 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014663 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c22e25-5694-4368-9462-5b84f2226eb4-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014675 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218629e5-8db6-4edd-91bc-179727432092-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014687 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014700 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4a673-85e7-412c-bd86-33d62b2be721-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014714 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014727 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.016005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.014741 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rv7q\" (UniqueName: \"kubernetes.io/projected/218629e5-8db6-4edd-91bc-179727432092-kube-api-access-9rv7q\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.018871 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.018839 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-kube-api-access-q8bjc" pod "83a4a673-85e7-412c-bd86-33d62b2be721" (UID: "83a4a673-85e7-412c-bd86-33d62b2be721"). InnerVolumeSpecName "kube-api-access-q8bjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:13.018976 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.018924 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-podinfo" pod "c27bd511-fd29-4697-8cd3-6d9d91ea9228" (UID: "c27bd511-fd29-4697-8cd3-6d9d91ea9228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:13.019038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.018981 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-istio-token" pod "01c22e25-5694-4368-9462-5b84f2226eb4" (UID: "01c22e25-5694-4368-9462-5b84f2226eb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:13.019107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.019086 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-credential-socket" pod "c27bd511-fd29-4697-8cd3-6d9d91ea9228" (UID: "c27bd511-fd29-4697-8cd3-6d9d91ea9228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.019157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.019090 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-token" pod "c27bd511-fd29-4697-8cd3-6d9d91ea9228" (UID: "c27bd511-fd29-4697-8cd3-6d9d91ea9228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:13.019208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.019165 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-istio-token" pod "83a4a673-85e7-412c-bd86-33d62b2be721" (UID: "83a4a673-85e7-412c-bd86-33d62b2be721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:13.019208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.019188 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istiod-ca-cert" pod "c27bd511-fd29-4697-8cd3-6d9d91ea9228" (UID: "c27bd511-fd29-4697-8cd3-6d9d91ea9228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:13.020268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.020238 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-kube-api-access-d2j48" pod "c27bd511-fd29-4697-8cd3-6d9d91ea9228" (UID: "c27bd511-fd29-4697-8cd3-6d9d91ea9228"). InnerVolumeSpecName "kube-api-access-d2j48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:13.023092 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.023069 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-kube-api-access-zjp8z" pod "01c22e25-5694-4368-9462-5b84f2226eb4" (UID: "01c22e25-5694-4368-9462-5b84f2226eb4"). InnerVolumeSpecName "kube-api-access-zjp8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:13.023202 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.023072 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-envoy" pod "01c22e25-5694-4368-9462-5b84f2226eb4" (UID: "01c22e25-5694-4368-9462-5b84f2226eb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.023275 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.023238 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83a4a673-85e7-412c-bd86-33d62b2be721-istio-podinfo" pod "83a4a673-85e7-412c-bd86-33d62b2be721" (UID: "83a4a673-85e7-412c-bd86-33d62b2be721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:13.023630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.023564 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01c22e25-5694-4368-9462-5b84f2226eb4-istio-podinfo" pod "01c22e25-5694-4368-9462-5b84f2226eb4" (UID: "01c22e25-5694-4368-9462-5b84f2226eb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:13.023984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.023962 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-envoy" pod "c27bd511-fd29-4697-8cd3-6d9d91ea9228" (UID: "c27bd511-fd29-4697-8cd3-6d9d91ea9228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.024171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.024149 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-envoy" pod "83a4a673-85e7-412c-bd86-33d62b2be721" (UID: "83a4a673-85e7-412c-bd86-33d62b2be721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:13.116321 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116288 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8bjc\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-kube-api-access-q8bjc\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116321 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116316 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjp8z\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-kube-api-access-zjp8z\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116321 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116328 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c22e25-5694-4368-9462-5b84f2226eb4-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116337 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c22e25-5694-4368-9462-5b84f2226eb4-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116346 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116359 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116371 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a4a673-85e7-412c-bd86-33d62b2be721-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116387 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116396 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c22e25-5694-4368-9462-5b84f2226eb4-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116405 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116413 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4a673-85e7-412c-bd86-33d62b2be721-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116424 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4a673-85e7-412c-bd86-33d62b2be721-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116437 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2j48\" (UniqueName: \"kubernetes.io/projected/c27bd511-fd29-4697-8cd3-6d9d91ea9228-kube-api-access-d2j48\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.116727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.116450 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27bd511-fd29-4697-8cd3-6d9d91ea9228-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:13.819426 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.819391 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" Oct 30 01:06:13.819426 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.819411 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8" event={"ID":"01c22e25-5694-4368-9462-5b84f2226eb4","Type":"ContainerDied","Data":"f60918405e9301931dff838a91ee2cd8a0d0b132bc5e72eccea1826aefa29853"} Oct 30 01:06:13.819714 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.819526 4927 scope.go:122] "RemoveContainer" containerID="1e8f32a8ec256628dcc592493a9680815c534411eab738ba38c4b83aafb55183" Oct 30 01:06:13.821137 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.821118 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" Oct 30 01:06:13.821259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.821148 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf" event={"ID":"c27bd511-fd29-4697-8cd3-6d9d91ea9228","Type":"ContainerDied","Data":"3031e49f9064a91c1efd343cb5287536fb0786ec677549e1b3db738fa0faccd6"} Oct 30 01:06:13.823005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.822979 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" Oct 30 01:06:13.823130 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.823001 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64" event={"ID":"83a4a673-85e7-412c-bd86-33d62b2be721","Type":"ContainerDied","Data":"1ae0491c4784ebbfafbd1e7e435e3413d48c6898ebd48c3db41eb1ed9bb9c225"} Oct 30 01:06:13.824862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.824833 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" event={"ID":"218629e5-8db6-4edd-91bc-179727432092","Type":"ContainerDied","Data":"2b53d2bf77b8d20722ee86a369d0406823a7f8141bb4a373590738dd028d4bf5"} Oct 30 01:06:13.824862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.824851 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p" Oct 30 01:06:13.833980 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.833962 4927 scope.go:122] "RemoveContainer" containerID="da4b60830c859647e114963fd70ddef0a29d34ad68902ed8873e9507452a76e9" Oct 30 01:06:13.845197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.845173 4927 scope.go:122] "RemoveContainer" containerID="de62e63e2098732cf7d2bc7429da8e0342c0c15c7ab3a1cd241c8d43b576c3ac" Oct 30 01:06:13.855997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.855980 4927 scope.go:122] "RemoveContainer" containerID="838f26bc22a2dd80a3c2346ed5671d8d6dab09e8c7f2afa10c13fa9cf16ffe8a" Oct 30 01:06:13.882722 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.882700 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64"] Oct 30 01:06:13.894020 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.894000 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-68l64"] Oct 30 01:06:13.927624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.927597 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p"] Oct 30 01:06:13.933692 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.933671 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-g2c9p"] Oct 30 01:06:13.967423 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.967394 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf"] Oct 30 01:06:13.972444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:13.972420 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-s5bpf"] Oct 30 01:06:14.005088 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:14.005061 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8"] Oct 30 01:06:14.015056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:14.015027 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-jk4x8"] Oct 30 01:06:15.574625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:15.574594 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" path="/var/lib/kubelet/pods/01c22e25-5694-4368-9462-5b84f2226eb4/volumes" Oct 30 01:06:15.575202 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:15.575187 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218629e5-8db6-4edd-91bc-179727432092" path="/var/lib/kubelet/pods/218629e5-8db6-4edd-91bc-179727432092/volumes" Oct 30 01:06:15.575735 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:15.575720 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" path="/var/lib/kubelet/pods/83a4a673-85e7-412c-bd86-33d62b2be721/volumes" Oct 30 01:06:15.576552 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:15.576537 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" path="/var/lib/kubelet/pods/c27bd511-fd29-4697-8cd3-6d9d91ea9228/volumes" Oct 30 01:06:28.036964 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.036924 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" podUID="eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" containerName="main" containerID="cri-o://bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258" gracePeriod=45 Oct 30 01:06:28.322904 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.322872 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:06:28.444338 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.444307 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tls-certs\" (UniqueName: \"kubernetes.io/secret/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tls-certs\") pod \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " Oct 30 01:06:28.444512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.444497 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-dshm\") pod \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " Oct 30 01:06:28.444607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.444565 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tmp-dir\") pod \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " Oct 30 01:06:28.444698 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.444618 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-model-cache\") pod \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " Oct 30 01:06:28.444698 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.444662 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-home\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-home\") pod \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " Oct 30 01:06:28.444885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.444855 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tmp-dir" pod "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" (UID: "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:28.445012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.444930 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-kube-api-access-p6gc7\" (UniqueName: \"kubernetes.io/projected/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-kube-api-access-p6gc7\") pod \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\" (UID: \"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8\") " Oct 30 01:06:28.445012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.444964 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-model-cache" pod "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" (UID: "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:28.445133 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.445016 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-home" pod "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" (UID: "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:28.445892 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.445866 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:28.445985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.445896 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:28.445985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.445912 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:28.446835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.446807 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tls-certs" pod "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" (UID: "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:28.447083 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.447056 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-dshm" pod "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" (UID: "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:28.447171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.447151 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-kube-api-access-p6gc7" pod "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" (UID: "eee589ce-f8fd-49b4-b1f7-406a26c8dfe8"). InnerVolumeSpecName "kube-api-access-p6gc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:28.548050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.547979 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6gc7\" (UniqueName: \"kubernetes.io/projected/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-kube-api-access-p6gc7\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:28.548050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.548010 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:28.548050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.548021 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:28.988985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.988944 4927 generic.go:356] "Generic (PLEG): container finished" podID="eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" containerID="bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258" exitCode=0 Oct 30 01:06:28.988985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.988981 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" Oct 30 01:06:28.989209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.989020 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" event={"ID":"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8","Type":"ContainerDied","Data":"bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258"} Oct 30 01:06:28.989209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.989063 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" event={"ID":"eee589ce-f8fd-49b4-b1f7-406a26c8dfe8","Type":"ContainerDied","Data":"f38fd14927125d91d24966505eb8ce26325ce9a053dbf19fa197c7ca8c71df0d"} Oct 30 01:06:28.989209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:28.989083 4927 scope.go:122] "RemoveContainer" containerID="bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258" Oct 30 01:06:29.002210 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:29.002181 4927 scope.go:122] "RemoveContainer" containerID="bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258" Oct 30 01:06:29.002481 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:06:29.002461 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258\": container with ID starting with bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258 not found: ID does not exist" containerID="bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258" Oct 30 01:06:29.002535 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:29.002491 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258"} err="failed to get container status \"bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258\": rpc error: code = NotFound desc = could not find container \"bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258\": container with ID starting with bb01346fdb43f5c346c212a196989d437c836160d7f2a809ab35fa7f6a709258 not found: ID does not exist" Oct 30 01:06:29.047469 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:29.047435 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm"] Oct 30 01:06:29.058526 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:29.058485 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm"] Oct 30 01:06:29.271376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:29.271288 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-43c02d/test-model-43c02d-kserve-8b557c447-tbmnm" podUID="eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.152:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:29.643672 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:29.643639 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" path="/var/lib/kubelet/pods/eee589ce-f8fd-49b4-b1f7-406a26c8dfe8/volumes" Oct 30 01:06:31.466821 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:31.466783 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg"] Oct 30 01:06:34.445080 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:06:34.444991 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:06:34.840347 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:34.840258 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh"] Oct 30 01:06:42.184443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:42.184407 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9"] Oct 30 01:06:42.217221 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:42.217189 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx"] Oct 30 01:06:46.494872 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.494822 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" podUID="21dc8a75-4b01-4f9c-af4a-4be8298c0b57" containerName="main" containerID="cri-o://cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3" gracePeriod=45 Oct 30 01:06:46.804050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.804017 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:06:46.896356 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.896313 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-model-cache\") pod \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " Oct 30 01:06:46.896558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.896430 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-kube-api-access-45jhm\" (UniqueName: \"kubernetes.io/projected/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-kube-api-access-45jhm\") pod \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " Oct 30 01:06:46.896558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.896512 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-home\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-home\") pod \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " Oct 30 01:06:46.896728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.896616 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-model-cache" pod "21dc8a75-4b01-4f9c-af4a-4be8298c0b57" (UID: "21dc8a75-4b01-4f9c-af4a-4be8298c0b57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:46.896728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.896625 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tmp-dir\") pod \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " Oct 30 01:06:46.896891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.896841 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tmp-dir" pod "21dc8a75-4b01-4f9c-af4a-4be8298c0b57" (UID: "21dc8a75-4b01-4f9c-af4a-4be8298c0b57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:46.898056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.897015 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-dshm\") pod \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " Oct 30 01:06:46.898056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.897097 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tls-certs\" (UniqueName: \"kubernetes.io/secret/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tls-certs\") pod \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\" (UID: \"21dc8a75-4b01-4f9c-af4a-4be8298c0b57\") " Oct 30 01:06:46.898056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.898010 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:46.898056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.898032 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:46.900050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.899270 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-kube-api-access-45jhm" pod "21dc8a75-4b01-4f9c-af4a-4be8298c0b57" (UID: "21dc8a75-4b01-4f9c-af4a-4be8298c0b57"). InnerVolumeSpecName "kube-api-access-45jhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:46.900050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.899358 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-home" pod "21dc8a75-4b01-4f9c-af4a-4be8298c0b57" (UID: "21dc8a75-4b01-4f9c-af4a-4be8298c0b57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:46.900050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.899780 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-dshm" pod "21dc8a75-4b01-4f9c-af4a-4be8298c0b57" (UID: "21dc8a75-4b01-4f9c-af4a-4be8298c0b57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:46.900521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.900492 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tls-certs" pod "21dc8a75-4b01-4f9c-af4a-4be8298c0b57" (UID: "21dc8a75-4b01-4f9c-af4a-4be8298c0b57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:46.999804 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.999764 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45jhm\" (UniqueName: \"kubernetes.io/projected/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-kube-api-access-45jhm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:46.999804 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.999798 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:46.999804 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.999811 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:47.000039 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:46.999823 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21dc8a75-4b01-4f9c-af4a-4be8298c0b57-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:47.185678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.185636 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" podUID="7d457c5a-4cc8-49e6-ac79-dafbd799684d" containerName="payload-processing" containerID="cri-o://b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d" gracePeriod=25 Oct 30 01:06:47.218473 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.218436 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" podUID="1a303900-2948-4ecd-8a56-04464434ee5d" containerName="payload-pre-processing" containerID="cri-o://d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3" gracePeriod=25 Oct 30 01:06:47.286641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.286607 4927 generic.go:356] "Generic (PLEG): container finished" podID="21dc8a75-4b01-4f9c-af4a-4be8298c0b57" containerID="cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3" exitCode=0 Oct 30 01:06:47.286808 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.286684 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" event={"ID":"21dc8a75-4b01-4f9c-af4a-4be8298c0b57","Type":"ContainerDied","Data":"cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3"} Oct 30 01:06:47.286808 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.286696 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" Oct 30 01:06:47.286808 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.286712 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg" event={"ID":"21dc8a75-4b01-4f9c-af4a-4be8298c0b57","Type":"ContainerDied","Data":"9f54e9c40415b1a0e70c00b311924de9101a0986449c46632f3f589a0c0ec3bd"} Oct 30 01:06:47.286808 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.286729 4927 scope.go:122] "RemoveContainer" containerID="cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3" Oct 30 01:06:47.299323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.299302 4927 scope.go:122] "RemoveContainer" containerID="cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3" Oct 30 01:06:47.299629 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:06:47.299607 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3\": container with ID starting with cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3 not found: ID does not exist" containerID="cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3" Oct 30 01:06:47.299679 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.299639 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3"} err="failed to get container status \"cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3\": rpc error: code = NotFound desc = could not find container \"cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3\": container with ID starting with cb1e500671538f20fc5e920dfa8460dbb899f35a600ad7a3efacad843d4db4e3 not found: ID does not exist" Oct 30 01:06:47.344757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.344721 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg"] Oct 30 01:06:47.357520 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.357487 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a07920/test-model-a07920-kserve-67f9d84db5-29chg"] Oct 30 01:06:47.645675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:47.643748 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21dc8a75-4b01-4f9c-af4a-4be8298c0b57" path="/var/lib/kubelet/pods/21dc8a75-4b01-4f9c-af4a-4be8298c0b57/volumes" Oct 30 01:06:49.787045 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:49.786782 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" podUID="83d079dd-5919-43c0-b623-f17e7b1bd38d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:49.980597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:49.965809 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.019660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:50.017689 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.139683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:50.118659 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" podUID="6a54e8b1-b885-49c6-b803-5cad96e73314" containerName="main" probeResult="failure" output="Get \"https://10.132.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.141193 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:50.141152 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" podUID="b99f3492-5f82-45bf-a645-9f28cad2a971" containerName="main" probeResult="failure" output="Get \"https://10.132.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.248699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:50.248649 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.365772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:50.365726 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" podUID="b99f3492-5f82-45bf-a645-9f28cad2a971" containerName="main" probeResult="failure" output="Get \"https://10.132.0.162:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.366149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:50.366015 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.389817 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:50.388833 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" podUID="7ea7d8b4-a022-4240-a420-b2443bca2a03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:50.566923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:50.566755 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" podUID="f086a848-42f7-4736-bd76-dfbffd894327" containerName="main" containerID="cri-o://835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44" gracePeriod=45 Oct 30 01:06:50.956702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:50.956672 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:06:51.056242 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.056201 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f086a848-42f7-4736-bd76-dfbffd894327-tls-certs\" (UniqueName: \"kubernetes.io/secret/f086a848-42f7-4736-bd76-dfbffd894327-tls-certs\") pod \"f086a848-42f7-4736-bd76-dfbffd894327\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " Oct 30 01:06:51.056428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.056274 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-dshm\") pod \"f086a848-42f7-4736-bd76-dfbffd894327\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " Oct 30 01:06:51.056428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.056350 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f086a848-42f7-4736-bd76-dfbffd894327-kube-api-access-n59z9\" (UniqueName: \"kubernetes.io/projected/f086a848-42f7-4736-bd76-dfbffd894327-kube-api-access-n59z9\") pod \"f086a848-42f7-4736-bd76-dfbffd894327\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " Oct 30 01:06:51.056558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.056499 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-home\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-home\") pod \"f086a848-42f7-4736-bd76-dfbffd894327\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " Oct 30 01:06:51.056703 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.056559 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-model-cache\") pod \"f086a848-42f7-4736-bd76-dfbffd894327\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " Oct 30 01:06:51.056798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.056777 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-tmp-dir\") pod \"f086a848-42f7-4736-bd76-dfbffd894327\" (UID: \"f086a848-42f7-4736-bd76-dfbffd894327\") " Oct 30 01:06:51.056947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.056918 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-home" pod "f086a848-42f7-4736-bd76-dfbffd894327" (UID: "f086a848-42f7-4736-bd76-dfbffd894327"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:51.057264 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.057124 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-model-cache" pod "f086a848-42f7-4736-bd76-dfbffd894327" (UID: "f086a848-42f7-4736-bd76-dfbffd894327"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:51.057380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.057292 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-tmp-dir" pod "f086a848-42f7-4736-bd76-dfbffd894327" (UID: "f086a848-42f7-4736-bd76-dfbffd894327"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:51.059614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.057705 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:51.059614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.057733 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:51.059614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.057746 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:51.059614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.059021 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f086a848-42f7-4736-bd76-dfbffd894327-tls-certs" pod "f086a848-42f7-4736-bd76-dfbffd894327" (UID: "f086a848-42f7-4736-bd76-dfbffd894327"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:51.059614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.059060 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-dshm" pod "f086a848-42f7-4736-bd76-dfbffd894327" (UID: "f086a848-42f7-4736-bd76-dfbffd894327"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:51.059914 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.059843 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f086a848-42f7-4736-bd76-dfbffd894327-kube-api-access-n59z9" pod "f086a848-42f7-4736-bd76-dfbffd894327" (UID: "f086a848-42f7-4736-bd76-dfbffd894327"). InnerVolumeSpecName "kube-api-access-n59z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:51.159335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.159298 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f086a848-42f7-4736-bd76-dfbffd894327-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:51.159335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.159334 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f086a848-42f7-4736-bd76-dfbffd894327-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:51.159554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.159349 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n59z9\" (UniqueName: \"kubernetes.io/projected/f086a848-42f7-4736-bd76-dfbffd894327-kube-api-access-n59z9\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:06:51.439996 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.439959 4927 generic.go:356] "Generic (PLEG): container finished" podID="f086a848-42f7-4736-bd76-dfbffd894327" containerID="835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44" exitCode=0 Oct 30 01:06:51.440209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.440022 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" event={"ID":"f086a848-42f7-4736-bd76-dfbffd894327","Type":"ContainerDied","Data":"835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44"} Oct 30 01:06:51.440209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.440076 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" event={"ID":"f086a848-42f7-4736-bd76-dfbffd894327","Type":"ContainerDied","Data":"43d6f18f931f9ab06bd4889b6cde66c923e18cff3b890ff0f178dc35a443019e"} Oct 30 01:06:51.440209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.440089 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" Oct 30 01:06:51.440209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.440094 4927 scope.go:122] "RemoveContainer" containerID="835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44" Oct 30 01:06:51.454093 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.454072 4927 scope.go:122] "RemoveContainer" containerID="835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44" Oct 30 01:06:51.454439 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:06:51.454410 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44\": container with ID starting with 835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44 not found: ID does not exist" containerID="835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44" Oct 30 01:06:51.454548 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.454451 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44"} err="failed to get container status \"835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44\": rpc error: code = NotFound desc = could not find container \"835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44\": container with ID starting with 835a2c45dfb0595b7e17f1a63202cd8c2bab402447c08d0d625e9e8437940c44 not found: ID does not exist" Oct 30 01:06:51.494351 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.494324 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh"] Oct 30 01:06:51.505961 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.505929 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh"] Oct 30 01:06:51.577001 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.576963 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f086a848-42f7-4736-bd76-dfbffd894327" path="/var/lib/kubelet/pods/f086a848-42f7-4736-bd76-dfbffd894327/volumes" Oct 30 01:06:51.789833 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:51.789742 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2a8b26/test-model-2a8b26-kserve-54867b88f7-x4cxh" podUID="f086a848-42f7-4736-bd76-dfbffd894327" containerName="main" probeResult="failure" output="Get \"https://10.132.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:56.149456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:56.149410 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" podUID="1a303900-2948-4ecd-8a56-04464434ee5d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:06:56.150029 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:06:56.149414 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" podUID="7d457c5a-4cc8-49e6-ac79-dafbd799684d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:07:01.587136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:01.587095 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv"] Oct 30 01:07:01.588191 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:01.588156 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" containerID="cri-o://b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07" gracePeriod=30 Oct 30 01:07:04.444467 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:07:04.444422 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:07:04.919585 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:04.919537 4927 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:07:04.919759 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:04.919614 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:07:06.149183 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:06.149144 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" podUID="7d457c5a-4cc8-49e6-ac79-dafbd799684d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:07:06.149183 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:06.149164 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" podUID="1a303900-2948-4ecd-8a56-04464434ee5d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:07:06.897539 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:06.897513 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 01:07:07.013338 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013304 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-credential-socket\") pod \"34591e8b-c9d0-4e40-892b-732be072817e\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " Oct 30 01:07:07.013522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013370 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34591e8b-c9d0-4e40-892b-732be072817e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34591e8b-c9d0-4e40-892b-732be072817e-istiod-ca-cert\") pod \"34591e8b-c9d0-4e40-892b-732be072817e\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " Oct 30 01:07:07.013522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013403 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-certs\") pod \"34591e8b-c9d0-4e40-892b-732be072817e\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " Oct 30 01:07:07.013699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013613 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-kube-api-access-z4nx7\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-kube-api-access-z4nx7\") pod \"34591e8b-c9d0-4e40-892b-732be072817e\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " Oct 30 01:07:07.013699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013674 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-istio-token\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-istio-token\") pod \"34591e8b-c9d0-4e40-892b-732be072817e\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " Oct 30 01:07:07.013814 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013692 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-credential-socket" pod "34591e8b-c9d0-4e40-892b-732be072817e" (UID: "34591e8b-c9d0-4e40-892b-732be072817e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:07.013814 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013731 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34591e8b-c9d0-4e40-892b-732be072817e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34591e8b-c9d0-4e40-892b-732be072817e-istio-podinfo\") pod \"34591e8b-c9d0-4e40-892b-732be072817e\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " Oct 30 01:07:07.013814 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013797 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-data\") pod \"34591e8b-c9d0-4e40-892b-732be072817e\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " Oct 30 01:07:07.013968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013836 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34591e8b-c9d0-4e40-892b-732be072817e-istiod-ca-cert" pod "34591e8b-c9d0-4e40-892b-732be072817e" (UID: "34591e8b-c9d0-4e40-892b-732be072817e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:07.013968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013860 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-certs" pod "34591e8b-c9d0-4e40-892b-732be072817e" (UID: "34591e8b-c9d0-4e40-892b-732be072817e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:07.013968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013902 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-socket\") pod \"34591e8b-c9d0-4e40-892b-732be072817e\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " Oct 30 01:07:07.014399 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.013992 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-envoy\") pod \"34591e8b-c9d0-4e40-892b-732be072817e\" (UID: \"34591e8b-c9d0-4e40-892b-732be072817e\") " Oct 30 01:07:07.014399 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.014170 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-data" pod "34591e8b-c9d0-4e40-892b-732be072817e" (UID: "34591e8b-c9d0-4e40-892b-732be072817e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:07.014399 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.014253 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-socket" pod "34591e8b-c9d0-4e40-892b-732be072817e" (UID: "34591e8b-c9d0-4e40-892b-732be072817e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:07.015952 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.015929 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.016055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.015953 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34591e8b-c9d0-4e40-892b-732be072817e-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.016055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.015966 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.016055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.015978 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.016055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.015990 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.016752 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.016721 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-kube-api-access-z4nx7" pod "34591e8b-c9d0-4e40-892b-732be072817e" (UID: "34591e8b-c9d0-4e40-892b-732be072817e"). InnerVolumeSpecName "kube-api-access-z4nx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:07.016850 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.016815 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34591e8b-c9d0-4e40-892b-732be072817e-istio-podinfo" pod "34591e8b-c9d0-4e40-892b-732be072817e" (UID: "34591e8b-c9d0-4e40-892b-732be072817e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:07.016900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.016844 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-istio-token" pod "34591e8b-c9d0-4e40-892b-732be072817e" (UID: "34591e8b-c9d0-4e40-892b-732be072817e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:07.016900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.016871 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-envoy" pod "34591e8b-c9d0-4e40-892b-732be072817e" (UID: "34591e8b-c9d0-4e40-892b-732be072817e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:07.132603 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.131907 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4nx7\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-kube-api-access-z4nx7\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.132603 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.131949 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34591e8b-c9d0-4e40-892b-732be072817e-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.132603 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.131966 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34591e8b-c9d0-4e40-892b-732be072817e-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.132603 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.131995 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34591e8b-c9d0-4e40-892b-732be072817e-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.494797 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.494757 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 01:07:07.560465 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.560430 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d457c5a-4cc8-49e6-ac79-dafbd799684d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d457c5a-4cc8-49e6-ac79-dafbd799684d-plugins-config-volume\") pod \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\" (UID: \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\") " Oct 30 01:07:07.564746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.560977 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d457c5a-4cc8-49e6-ac79-dafbd799684d-kube-api-access-brdj2\" (UniqueName: \"kubernetes.io/projected/7d457c5a-4cc8-49e6-ac79-dafbd799684d-kube-api-access-brdj2\") pod \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\" (UID: \"7d457c5a-4cc8-49e6-ac79-dafbd799684d\") " Oct 30 01:07:07.566399 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.566087 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d457c5a-4cc8-49e6-ac79-dafbd799684d-plugins-config-volume" pod "7d457c5a-4cc8-49e6-ac79-dafbd799684d" (UID: "7d457c5a-4cc8-49e6-ac79-dafbd799684d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:07.573776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.573383 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d457c5a-4cc8-49e6-ac79-dafbd799684d-kube-api-access-brdj2" pod "7d457c5a-4cc8-49e6-ac79-dafbd799684d" (UID: "7d457c5a-4cc8-49e6-ac79-dafbd799684d"). InnerVolumeSpecName "kube-api-access-brdj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:07.591310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.591275 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 01:07:07.667061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.666988 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a303900-2948-4ecd-8a56-04464434ee5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a303900-2948-4ecd-8a56-04464434ee5d-plugins-config-volume\") pod \"1a303900-2948-4ecd-8a56-04464434ee5d\" (UID: \"1a303900-2948-4ecd-8a56-04464434ee5d\") " Oct 30 01:07:07.667289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.667264 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a303900-2948-4ecd-8a56-04464434ee5d-kube-api-access-2hdwd\" (UniqueName: \"kubernetes.io/projected/1a303900-2948-4ecd-8a56-04464434ee5d-kube-api-access-2hdwd\") pod \"1a303900-2948-4ecd-8a56-04464434ee5d\" (UID: \"1a303900-2948-4ecd-8a56-04464434ee5d\") " Oct 30 01:07:07.667438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.667415 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a303900-2948-4ecd-8a56-04464434ee5d-plugins-config-volume" pod "1a303900-2948-4ecd-8a56-04464434ee5d" (UID: "1a303900-2948-4ecd-8a56-04464434ee5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:07.668702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.668674 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brdj2\" (UniqueName: \"kubernetes.io/projected/7d457c5a-4cc8-49e6-ac79-dafbd799684d-kube-api-access-brdj2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.668831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.668707 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d457c5a-4cc8-49e6-ac79-dafbd799684d-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.668831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.668725 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a303900-2948-4ecd-8a56-04464434ee5d-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.682993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.682957 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a303900-2948-4ecd-8a56-04464434ee5d-kube-api-access-2hdwd" pod "1a303900-2948-4ecd-8a56-04464434ee5d" (UID: "1a303900-2948-4ecd-8a56-04464434ee5d"). InnerVolumeSpecName "kube-api-access-2hdwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:07.698130 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.698098 4927 generic.go:356] "Generic (PLEG): container finished" podID="34591e8b-c9d0-4e40-892b-732be072817e" containerID="b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07" exitCode=0 Oct 30 01:07:07.698259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.698200 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" Oct 30 01:07:07.698259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.698212 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" event={"ID":"34591e8b-c9d0-4e40-892b-732be072817e","Type":"ContainerDied","Data":"b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07"} Oct 30 01:07:07.698386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.698301 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv" event={"ID":"34591e8b-c9d0-4e40-892b-732be072817e","Type":"ContainerDied","Data":"bf0a4305a38613a4269b9e0ef208362263264e20a44a9b445f4bc59ce513abe0"} Oct 30 01:07:07.698386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.698332 4927 scope.go:122] "RemoveContainer" containerID="b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07" Oct 30 01:07:07.700800 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.700773 4927 generic.go:356] "Generic (PLEG): container finished" podID="7d457c5a-4cc8-49e6-ac79-dafbd799684d" containerID="b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d" exitCode=0 Oct 30 01:07:07.700909 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.700825 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" Oct 30 01:07:07.700909 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.700873 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" event={"ID":"7d457c5a-4cc8-49e6-ac79-dafbd799684d","Type":"ContainerDied","Data":"b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d"} Oct 30 01:07:07.701044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.700911 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9" event={"ID":"7d457c5a-4cc8-49e6-ac79-dafbd799684d","Type":"ContainerDied","Data":"330f7b49b60af28131f97d17a1bc5d622b95614e6b30f30b9bd079e12c1806ff"} Oct 30 01:07:07.703819 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.703742 4927 generic.go:356] "Generic (PLEG): container finished" podID="1a303900-2948-4ecd-8a56-04464434ee5d" containerID="d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3" exitCode=0 Oct 30 01:07:07.703948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.703820 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" Oct 30 01:07:07.704116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.704031 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" event={"ID":"1a303900-2948-4ecd-8a56-04464434ee5d","Type":"ContainerDied","Data":"d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3"} Oct 30 01:07:07.704116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.704086 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx" event={"ID":"1a303900-2948-4ecd-8a56-04464434ee5d","Type":"ContainerDied","Data":"f8acf455184b2b5fea4e77614dae04d4f391fff5876dcce47f97ab6be7a2afc6"} Oct 30 01:07:07.715393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.715289 4927 scope.go:122] "RemoveContainer" containerID="864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf" Oct 30 01:07:07.725819 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.725802 4927 scope.go:122] "RemoveContainer" containerID="b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07" Oct 30 01:07:07.726089 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:07:07.726071 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07\": container with ID starting with b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07 not found: ID does not exist" containerID="b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07" Oct 30 01:07:07.726156 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.726095 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07"} err="failed to get container status \"b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07\": rpc error: code = NotFound desc = could not find container \"b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07\": container with ID starting with b317d99b37f86fa5a0263c8f0ff68d1dc69911758eb97d45978131913d1c1f07 not found: ID does not exist" Oct 30 01:07:07.726156 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.726111 4927 scope.go:122] "RemoveContainer" containerID="864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf" Oct 30 01:07:07.726328 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:07:07.726311 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf\": container with ID starting with 864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf not found: ID does not exist" containerID="864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf" Oct 30 01:07:07.726365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.726332 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf"} err="failed to get container status \"864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf\": rpc error: code = NotFound desc = could not find container \"864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf\": container with ID starting with 864a2264e51dff06c466c5b47e1ace769c89323a6dcabd7a425bf2025fe53dbf not found: ID does not exist" Oct 30 01:07:07.726365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.726345 4927 scope.go:122] "RemoveContainer" containerID="b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d" Oct 30 01:07:07.736785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.736768 4927 scope.go:122] "RemoveContainer" containerID="b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d" Oct 30 01:07:07.737051 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:07:07.737017 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d\": container with ID starting with b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d not found: ID does not exist" containerID="b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d" Oct 30 01:07:07.737149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.737054 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d"} err="failed to get container status \"b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d\": rpc error: code = NotFound desc = could not find container \"b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d\": container with ID starting with b093114f7697660a23807da04b2d3441abbff1f4916ec0dca148d18d02f9697d not found: ID does not exist" Oct 30 01:07:07.737149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.737070 4927 scope.go:122] "RemoveContainer" containerID="d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3" Oct 30 01:07:07.748772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.748748 4927 scope.go:122] "RemoveContainer" containerID="d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3" Oct 30 01:07:07.749093 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:07:07.749070 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3\": container with ID starting with d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3 not found: ID does not exist" containerID="d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3" Oct 30 01:07:07.749179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.749104 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3"} err="failed to get container status \"d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3\": rpc error: code = NotFound desc = could not find container \"d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3\": container with ID starting with d88d9ff1a8dd20570c65522704562d8fdc28687df7a5b20ee60743d22de29dc3 not found: ID does not exist" Oct 30 01:07:07.751543 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.751522 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9"] Oct 30 01:07:07.757582 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.757546 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f4f9b-57b8fc4544-pbpc9"] Oct 30 01:07:07.771106 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.771076 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hdwd\" (UniqueName: \"kubernetes.io/projected/1a303900-2948-4ecd-8a56-04464434ee5d-kube-api-access-2hdwd\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:07.791814 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.791785 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv"] Oct 30 01:07:07.797259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.797232 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-6cggv"] Oct 30 01:07:07.843300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.843269 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx"] Oct 30 01:07:07.848198 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:07.848173 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f4f9b-794b475b4c-hkwrx"] Oct 30 01:07:09.229595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:09.212914 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f"] Oct 30 01:07:09.229595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:09.213122 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" containerID="cri-o://cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927" gracePeriod=30 Oct 30 01:07:09.593520 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:09.593445 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a303900-2948-4ecd-8a56-04464434ee5d" path="/var/lib/kubelet/pods/1a303900-2948-4ecd-8a56-04464434ee5d/volumes" Oct 30 01:07:09.594182 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:09.594161 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34591e8b-c9d0-4e40-892b-732be072817e" path="/var/lib/kubelet/pods/34591e8b-c9d0-4e40-892b-732be072817e/volumes" Oct 30 01:07:09.594723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:09.594702 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d457c5a-4cc8-49e6-ac79-dafbd799684d" path="/var/lib/kubelet/pods/7d457c5a-4cc8-49e6-ac79-dafbd799684d/volumes" Oct 30 01:07:13.039507 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.039470 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5"] Oct 30 01:07:13.040342 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040327 4927 cpu_manager.go:414] "containerMap: removing container" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" Oct 30 01:07:13.040342 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040343 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040356 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f086a848-42f7-4736-bd76-dfbffd894327" containerName="main" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040361 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f086a848-42f7-4736-bd76-dfbffd894327" containerName="main" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040369 4927 cpu_manager.go:414] "containerMap: removing container" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040374 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040387 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040391 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040408 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7d457c5a-4cc8-49e6-ac79-dafbd799684d" containerName="payload-processing" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040412 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d457c5a-4cc8-49e6-ac79-dafbd799684d" containerName="payload-processing" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040433 4927 cpu_manager.go:414] "containerMap: removing container" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040438 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040445 4927 cpu_manager.go:414] "containerMap: removing container" podUID="21dc8a75-4b01-4f9c-af4a-4be8298c0b57" containerName="main" Oct 30 01:07:13.040448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040450 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dc8a75-4b01-4f9c-af4a-4be8298c0b57" containerName="main" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040459 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1a303900-2948-4ecd-8a56-04464434ee5d" containerName="payload-pre-processing" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040464 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a303900-2948-4ecd-8a56-04464434ee5d" containerName="payload-pre-processing" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040474 4927 cpu_manager.go:414] "containerMap: removing container" podUID="eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" containerName="main" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040479 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" containerName="main" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040491 4927 cpu_manager.go:414] "containerMap: removing container" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040495 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040643 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="218629e5-8db6-4edd-91bc-179727432092" containerName="istio-proxy" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040652 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dc8a75-4b01-4f9c-af4a-4be8298c0b57" containerName="main" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040658 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a303900-2948-4ecd-8a56-04464434ee5d" containerName="payload-pre-processing" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040664 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee589ce-f8fd-49b4-b1f7-406a26c8dfe8" containerName="main" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040671 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c22e25-5694-4368-9462-5b84f2226eb4" containerName="istio-proxy" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040680 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040687 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a4a673-85e7-412c-bd86-33d62b2be721" containerName="istio-proxy" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040693 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f086a848-42f7-4736-bd76-dfbffd894327" containerName="main" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040699 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040706 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27bd511-fd29-4697-8cd3-6d9d91ea9228" containerName="istio-proxy" Oct 30 01:07:13.040847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.040713 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d457c5a-4cc8-49e6-ac79-dafbd799684d" containerName="payload-processing" Oct 30 01:07:13.046644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.046624 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.055648 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.055620 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n"] Oct 30 01:07:13.056794 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.056773 4927 cpu_manager.go:414] "containerMap: removing container" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" Oct 30 01:07:13.056794 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.056793 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34591e8b-c9d0-4e40-892b-732be072817e" containerName="istio-proxy" Oct 30 01:07:13.061242 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.061223 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.061325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.061224 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh"] Oct 30 01:07:13.062784 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.062765 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ae06fe-openshift-default-dockercfg-ccc5k\"" Oct 30 01:07:13.066839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.066816 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.072050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.072027 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5"] Oct 30 01:07:13.102865 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.102829 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n"] Oct 30 01:07:13.103913 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.103885 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.103986 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.103935 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.104030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.103983 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.104030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104008 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.104103 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104046 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.104103 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104083 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.104175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104110 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbj7\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-kube-api-access-scbj7\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.104175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104166 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.104237 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104203 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42m9\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-kube-api-access-x42m9\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.104290 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104273 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.104353 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104337 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.104396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104366 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.104438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104420 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.104476 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104445 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.104537 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104520 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.104608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104557 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.104661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104633 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.104716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104698 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.104776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104762 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.104821 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104792 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffb6932-005f-4829-a8c0-9ff8801be789-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.104857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104829 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.104895 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104854 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.104933 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.104912 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.105050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.105033 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.105096 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.105072 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8bf\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-kube-api-access-lq8bf\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.105140 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.105113 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.105176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.105159 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41bd266-1e41-448a-bf1c-ca43487dd741-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.109395 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.109369 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh"] Oct 30 01:07:13.229984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.229932 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.241927 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.235318 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.242264 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.242240 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.265494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.262355 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.265494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.262696 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffb6932-005f-4829-a8c0-9ff8801be789-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.286142 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffb6932-005f-4829-a8c0-9ff8801be789-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.286397 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.286468 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.286930 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287005 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287087 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287235 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287284 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8bf\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-kube-api-access-lq8bf\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287331 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287390 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41bd266-1e41-448a-bf1c-ca43487dd741-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287439 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287476 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287538 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287562 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287647 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287708 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287752 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbj7\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-kube-api-access-scbj7\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287789 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287846 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42m9\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-kube-api-access-x42m9\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.287951 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.288053 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.288080 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.288158 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.288184 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.288301 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.288338 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.288927 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.289548 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41bd266-1e41-448a-bf1c-ca43487dd741-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.289834 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.290145 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.310687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.290405 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.327662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.323562 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.327662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.324652 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.327662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.325192 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.327662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.325467 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.327662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.325727 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.327662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.325984 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.327662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.326750 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.328617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.328591 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.342231 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.341528 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.342413 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.341912 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.354602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.343179 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.368700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.355126 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.368700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.355348 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.387068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.372824 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbj7\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-kube-api-access-scbj7\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.389256 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.389215 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.417390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.411427 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8bf\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-kube-api-access-lq8bf\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.417390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.412495 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.417390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.412660 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42m9\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-kube-api-access-x42m9\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:13.664265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.664228 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:13.684144 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.684112 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:13.697644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:13.697617 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:14.022460 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.022196 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5"] Oct 30 01:07:14.024758 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:07:14.024462 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffb6932_005f_4829_a8c0_9ff8801be789.slice/crio-e8cc0a20ba7287312a7acbd56a35739032da43ef8d983a3eb22e502cf2e11400 WatchSource:0}: Error finding container e8cc0a20ba7287312a7acbd56a35739032da43ef8d983a3eb22e502cf2e11400: Status 404 returned error can't find the container with id e8cc0a20ba7287312a7acbd56a35739032da43ef8d983a3eb22e502cf2e11400 Oct 30 01:07:14.025067 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:07:14.025043 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41bd266_1e41_448a_bf1c_ca43487dd741.slice/crio-8f2b7f1264532cda08e537f2de4104d65effa74d0b95343635536160d86dea68 WatchSource:0}: Error finding container 8f2b7f1264532cda08e537f2de4104d65effa74d0b95343635536160d86dea68: Status 404 returned error can't find the container with id 8f2b7f1264532cda08e537f2de4104d65effa74d0b95343635536160d86dea68 Oct 30 01:07:14.028970 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.028930 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:14.029078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.029044 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:14.029134 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.029097 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:14.030176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.030143 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:14.030268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.030205 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:14.030268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.030239 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:14.031038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.031004 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n"] Oct 30 01:07:14.063473 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.063265 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh"] Oct 30 01:07:14.066493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.066459 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:14.066668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.066541 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:14.066668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.066607 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:14.675026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.674997 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 01:07:14.743238 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.743196 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2264a783-695c-45cd-953b-9cda029a572b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264a783-695c-45cd-953b-9cda029a572b-istio-podinfo\") pod \"2264a783-695c-45cd-953b-9cda029a572b\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " Oct 30 01:07:14.743443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.743284 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2264a783-695c-45cd-953b-9cda029a572b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264a783-695c-45cd-953b-9cda029a572b-istiod-ca-cert\") pod \"2264a783-695c-45cd-953b-9cda029a572b\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " Oct 30 01:07:14.743443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.743370 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-credential-socket\") pod \"2264a783-695c-45cd-953b-9cda029a572b\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " Oct 30 01:07:14.744233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.744194 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-credential-socket" pod "2264a783-695c-45cd-953b-9cda029a572b" (UID: "2264a783-695c-45cd-953b-9cda029a572b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:14.744515 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.744496 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2264a783-695c-45cd-953b-9cda029a572b-istiod-ca-cert" pod "2264a783-695c-45cd-953b-9cda029a572b" (UID: "2264a783-695c-45cd-953b-9cda029a572b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:14.749491 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.748975 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2264a783-695c-45cd-953b-9cda029a572b-istio-podinfo" pod "2264a783-695c-45cd-953b-9cda029a572b" (UID: "2264a783-695c-45cd-953b-9cda029a572b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:14.847593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.846949 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-istio-token\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-istio-token\") pod \"2264a783-695c-45cd-953b-9cda029a572b\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " Oct 30 01:07:14.847593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.846995 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-socket\") pod \"2264a783-695c-45cd-953b-9cda029a572b\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " Oct 30 01:07:14.847593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.847115 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-envoy\") pod \"2264a783-695c-45cd-953b-9cda029a572b\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " Oct 30 01:07:14.847593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.847158 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-certs\") pod \"2264a783-695c-45cd-953b-9cda029a572b\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " Oct 30 01:07:14.847593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.847196 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-data\") pod \"2264a783-695c-45cd-953b-9cda029a572b\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " Oct 30 01:07:14.847593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.847509 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-certs" pod "2264a783-695c-45cd-953b-9cda029a572b" (UID: "2264a783-695c-45cd-953b-9cda029a572b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:14.848063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.847753 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-data" pod "2264a783-695c-45cd-953b-9cda029a572b" (UID: "2264a783-695c-45cd-953b-9cda029a572b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:14.848063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.847793 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-socket" pod "2264a783-695c-45cd-953b-9cda029a572b" (UID: "2264a783-695c-45cd-953b-9cda029a572b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:14.848063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.847861 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-kube-api-access-2w4rl\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-kube-api-access-2w4rl\") pod \"2264a783-695c-45cd-953b-9cda029a572b\" (UID: \"2264a783-695c-45cd-953b-9cda029a572b\") " Oct 30 01:07:14.857600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.848538 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:14.857600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.848555 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:14.857600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.848564 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264a783-695c-45cd-953b-9cda029a572b-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:14.857600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.848593 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264a783-695c-45cd-953b-9cda029a572b-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:14.857600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.848605 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:14.857600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.848616 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:14.913149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.911611 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-istio-token" pod "2264a783-695c-45cd-953b-9cda029a572b" (UID: "2264a783-695c-45cd-953b-9cda029a572b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:14.957597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.955783 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-kube-api-access-2w4rl" pod "2264a783-695c-45cd-953b-9cda029a572b" (UID: "2264a783-695c-45cd-953b-9cda029a572b"). InnerVolumeSpecName "kube-api-access-2w4rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:14.957810 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.957627 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:14.957810 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:14.957658 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w4rl\" (UniqueName: \"kubernetes.io/projected/2264a783-695c-45cd-953b-9cda029a572b-kube-api-access-2w4rl\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:15.047586 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.047264 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-envoy" pod "2264a783-695c-45cd-953b-9cda029a572b" (UID: "2264a783-695c-45cd-953b-9cda029a572b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:15.064596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.063646 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" event={"ID":"d41bd266-1e41-448a-bf1c-ca43487dd741","Type":"ContainerStarted","Data":"458f492dbdfe48789f98ebda7d0c5bd456be466b6343b4add8b2e03cad25bbfd"} Oct 30 01:07:15.064596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.063806 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" event={"ID":"d41bd266-1e41-448a-bf1c-ca43487dd741","Type":"ContainerStarted","Data":"8f2b7f1264532cda08e537f2de4104d65effa74d0b95343635536160d86dea68"} Oct 30 01:07:15.092807 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.076148 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" event={"ID":"4a43e4d1-4dad-4473-82e8-3c51b2bb8753","Type":"ContainerStarted","Data":"f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28"} Oct 30 01:07:15.092807 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.076220 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" event={"ID":"4a43e4d1-4dad-4473-82e8-3c51b2bb8753","Type":"ContainerStarted","Data":"f1e84b4a70134f1a32881f9b4b899d93997ce0f2fe685686efca2c518197dd0c"} Oct 30 01:07:15.092807 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.088705 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264a783-695c-45cd-953b-9cda029a572b-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:15.149652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.140781 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" podStartSLOduration=2.14076218 podStartE2EDuration="2.14076218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:15.138150394 +0000 UTC m=+1973.710076618" watchObservedRunningTime="2026-10-30 01:07:15.14076218 +0000 UTC m=+1973.712688403" Oct 30 01:07:15.149652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.142220 4927 generic.go:356] "Generic (PLEG): container finished" podID="2264a783-695c-45cd-953b-9cda029a572b" containerID="cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927" exitCode=0 Oct 30 01:07:15.149652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.142385 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" event={"ID":"2264a783-695c-45cd-953b-9cda029a572b","Type":"ContainerDied","Data":"cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927"} Oct 30 01:07:15.149652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.142457 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" event={"ID":"2264a783-695c-45cd-953b-9cda029a572b","Type":"ContainerDied","Data":"9a206a427664db51f1f045e0dab8d5646eb7ce965f78dff348f7b2bf8399263e"} Oct 30 01:07:15.149652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.142488 4927 scope.go:122] "RemoveContainer" containerID="cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927" Oct 30 01:07:15.149652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.142741 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f" Oct 30 01:07:15.244294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.242727 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" event={"ID":"1ffb6932-005f-4829-a8c0-9ff8801be789","Type":"ContainerStarted","Data":"aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077"} Oct 30 01:07:15.244294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.242805 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" event={"ID":"1ffb6932-005f-4829-a8c0-9ff8801be789","Type":"ContainerStarted","Data":"e8cc0a20ba7287312a7acbd56a35739032da43ef8d983a3eb22e502cf2e11400"} Oct 30 01:07:15.264058 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.263555 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" podStartSLOduration=2.263537513 podStartE2EDuration="2.263537513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:15.191228863 +0000 UTC m=+1973.763155085" watchObservedRunningTime="2026-10-30 01:07:15.263537513 +0000 UTC m=+1973.835463740" Oct 30 01:07:15.467934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.467805 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" podStartSLOduration=2.4677807339999998 podStartE2EDuration="2.467780734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:15.294001011 +0000 UTC m=+1973.865927234" watchObservedRunningTime="2026-10-30 01:07:15.467780734 +0000 UTC m=+1974.039706960" Oct 30 01:07:15.567344 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.566767 4927 scope.go:122] "RemoveContainer" containerID="cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927" Oct 30 01:07:15.567344 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:07:15.567226 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927\": container with ID starting with cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927 not found: ID does not exist" containerID="cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927" Oct 30 01:07:15.567344 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.567263 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927"} err="failed to get container status \"cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927\": rpc error: code = NotFound desc = could not find container \"cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927\": container with ID starting with cd588257e8ca793b5bf2172a6c3eb2b8f69c3eca7419895fb76888e373d72927 not found: ID does not exist" Oct 30 01:07:15.649052 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.649021 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f"] Oct 30 01:07:15.651242 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.651213 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-ldp9f"] Oct 30 01:07:15.665017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.664981 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:15.667279 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.667245 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:07:15.667412 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.667316 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:07:15.706246 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.705270 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:15.706246 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.705608 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:15.730651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.730287 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:07:15.730651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.730363 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:07:15.730906 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.730676 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 01:07:15.730906 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.730723 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 01:07:15.989795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.981649 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b"] Oct 30 01:07:15.989795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.983102 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" Oct 30 01:07:15.989795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.983127 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" Oct 30 01:07:15.989795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:15.983460 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2264a783-695c-45cd-953b-9cda029a572b" containerName="istio-proxy" Oct 30 01:07:16.023653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.021952 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.059596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.050953 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b"] Oct 30 01:07:16.109827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.088785 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.109827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.088875 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.109827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.088927 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.109827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.089005 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.109827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.089102 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.109827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.089165 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.109827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.089223 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.109827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.092724 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4378ab-80c5-42d8-a817-8c11650d094b-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.109827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.092787 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5sq\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-kube-api-access-8x5sq\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.258727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.250471 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.258727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.250539 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.264705 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.259072 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.271040 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.270886 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4378ab-80c5-42d8-a817-8c11650d094b-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.271040 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.270938 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5sq\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-kube-api-access-8x5sq\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.271384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.271187 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.271384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.271260 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.271384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.271291 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.271384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.271370 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.271759 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.271476 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.272357 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.272329 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.273522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.273479 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4378ab-80c5-42d8-a817-8c11650d094b-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.273810 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.273787 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.332638 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.330543 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.362026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.361993 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5sq\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-kube-api-access-8x5sq\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.373040 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.371628 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.417790 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.373853 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.417790 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.386873 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.471792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.471356 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:16.664792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.664753 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:07:16.664953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.664815 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:07:16.714590 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.712962 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:07:16.714590 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.713023 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:07:16.714590 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.713618 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 01:07:16.714590 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:16.713656 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 01:07:17.737632 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.734789 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:07:17.737632 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.734855 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:07:17.763105 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.742132 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:07:17.763105 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.742197 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:07:17.763105 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.742277 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 01:07:17.763105 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.742298 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 01:07:17.858593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.855947 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2264a783-695c-45cd-953b-9cda029a572b" path="/var/lib/kubelet/pods/2264a783-695c-45cd-953b-9cda029a572b/volumes" Oct 30 01:07:17.858593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.856491 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv"] Oct 30 01:07:17.874691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.874651 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" podUID="faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:17.911216 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:17.911171 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" containerID="cri-o://6d1b139e4634327f1f4ccf494c5f3182f858e448eb8ec7fff9e74576fc6513c8" gracePeriod=30 Oct 30 01:07:18.165165 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.164641 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" podUID="365d784b-a986-49a5-ba10-93a80bdf2630" containerName="main" probeResult="failure" output="Get \"https://10.132.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:18.170818 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:07:18.168819 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4378ab_80c5_42d8_a817_8c11650d094b.slice/crio-747ea1e79abfd5cb0323add57102682aa9ddd76cd636172e550818b67216ccea WatchSource:0}: Error finding container 747ea1e79abfd5cb0323add57102682aa9ddd76cd636172e550818b67216ccea: Status 404 returned error can't find the container with id 747ea1e79abfd5cb0323add57102682aa9ddd76cd636172e550818b67216ccea Oct 30 01:07:18.217793 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.213669 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" podUID="b99f3492-5f82-45bf-a645-9f28cad2a971" containerName="main" probeResult="failure" output="Get \"https://10.132.0.162:8000/ready\": context deadline exceeded" Oct 30 01:07:18.249679 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.246795 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" podUID="6a54e8b1-b885-49c6-b803-5cad96e73314" containerName="main" probeResult="failure" output="Get \"https://10.132.0.161:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:18.249679 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.248112 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:18.249679 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.248223 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:18.249679 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.248264 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:18.275594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.250375 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b"] Oct 30 01:07:18.275594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.252796 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:18.736595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.725485 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 01:07:18.766870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.764375 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:07:18.766870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.764440 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:07:18.777601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.767738 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 01:07:18.869169 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.869119 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:07:18.869337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:18.869193 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:07:19.064109 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:19.064006 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" event={"ID":"9f4378ab-80c5-42d8-a817-8c11650d094b","Type":"ContainerStarted","Data":"747ea1e79abfd5cb0323add57102682aa9ddd76cd636172e550818b67216ccea"} Oct 30 01:07:19.671597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:19.670325 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:07:19.671597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:19.670390 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:07:20.033647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.020244 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:07:20.033647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.020309 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:07:20.033647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.020398 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 01:07:20.033647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.020422 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 01:07:20.094270 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.094231 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" event={"ID":"9f4378ab-80c5-42d8-a817-8c11650d094b","Type":"ContainerStarted","Data":"36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946"} Oct 30 01:07:20.279593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.275764 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" podStartSLOduration=5.275743146 podStartE2EDuration="5.275743146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:20.201986362 +0000 UTC m=+1978.773912585" watchObservedRunningTime="2026-10-30 01:07:20.275743146 +0000 UTC m=+1978.847669369" Oct 30 01:07:20.432596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.401185 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" podUID="7ea7d8b4-a022-4240-a420-b2443bca2a03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:20.625659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.608946 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:20.626661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.626556 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:07:20.627608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.626859 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 01:07:20.710648 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.686915 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 01:07:20.719705 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.719653 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 01:07:20.799740 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.779986 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:07:20.799740 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.780052 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:07:20.799740 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.795723 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:07:20.799740 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.795778 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:07:20.799740 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.795923 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z"] Oct 30 01:07:20.946365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:20.946322 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" podUID="faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded" Oct 30 01:07:21.239035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.238999 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z"] Oct 30 01:07:21.239736 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.239712 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.276029 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.275976 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a07920-openshift-default-dockercfg-pc9lh\"" Oct 30 01:07:21.357473 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.356767 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:21.414618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.402640 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" podUID="7ea7d8b4-a022-4240-a420-b2443bca2a03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:07:21.414618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.403353 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:07:21.491975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.471555 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef083d76-7f00-49e0-b7d4-657f48389486-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.491975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.471635 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.491975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.471692 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.491975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.471742 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.491975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.471796 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.491975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.471932 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef083d76-7f00-49e0-b7d4-657f48389486-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.491975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.471988 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqd5\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-kube-api-access-ztqd5\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.491975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.472029 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.491975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.472115 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.495984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.495953 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:07:21.785650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.763292 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 01:07:21.785650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.504654 4927 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:07:21.785650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.765379 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:07:21.785650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.775076 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.785650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.775168 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.785650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.775223 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.785650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.775313 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.822745 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.822700 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef083d76-7f00-49e0-b7d4-657f48389486-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.823096 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.823074 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqd5\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-kube-api-access-ztqd5\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.823228 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.823214 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.823410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.823395 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.823943 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.823919 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef083d76-7f00-49e0-b7d4-657f48389486-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.849016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.848976 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:07:21.870601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.870287 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:07:21.870601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.849425 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:07:21.870601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.870523 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:07:21.890499 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.890459 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.941944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.941255 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.948060 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.948024 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:21.966525 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:21.966272 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef083d76-7f00-49e0-b7d4-657f48389486-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:22.025596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.008321 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:22.043389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.043300 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:07:22.045115 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.045083 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef083d76-7f00-49e0-b7d4-657f48389486-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:22.163050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.163011 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:22.352233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.352153 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqd5\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-kube-api-access-ztqd5\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:22.360561 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.360519 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:22.457625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.447114 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:22.514713 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.479199 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:22.564826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.515345 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:07:22.564826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.515409 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 01:07:22.564826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.546450 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:07:22.566444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.566406 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a07920-openshift-default-dockercfg-pc9lh\"" Oct 30 01:07:22.574590 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.569526 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:22.706597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.704433 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:22.710594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.709879 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:22.726654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.713789 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:07:22.726654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.718835 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:22.736600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.728717 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:22.736600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:22.736529 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:07:23.113984 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:07:23.113953 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef083d76_7f00_49e0_b7d4_657f48389486.slice/crio-908cd4a3efb7e3c61e7f5e0d38d15ce4cd09f90e7c2730696ac783a82fee0b9e WatchSource:0}: Error finding container 908cd4a3efb7e3c61e7f5e0d38d15ce4cd09f90e7c2730696ac783a82fee0b9e: Status 404 returned error can't find the container with id 908cd4a3efb7e3c61e7f5e0d38d15ce4cd09f90e7c2730696ac783a82fee0b9e Oct 30 01:07:23.117716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.116559 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z"] Oct 30 01:07:23.117716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.117223 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:23.117716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.117294 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:23.117716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.117329 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:23.499934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.472908 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:07:23.499934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.472966 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 01:07:23.519596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.500983 4927 generic.go:356] "Generic (PLEG): container finished" podID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerID="6d1b139e4634327f1f4ccf494c5f3182f858e448eb8ec7fff9e74576fc6513c8" exitCode=0 Oct 30 01:07:23.519596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.501193 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" event={"ID":"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a","Type":"ContainerDied","Data":"6d1b139e4634327f1f4ccf494c5f3182f858e448eb8ec7fff9e74576fc6513c8"} Oct 30 01:07:23.519596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.501299 4927 scope.go:122] "RemoveContainer" containerID="4e5185d39cc1e4e1df84d34d9df7b2d44231be8aa577532b9a7208538475eee1" Oct 30 01:07:23.541003 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.539560 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" event={"ID":"ef083d76-7f00-49e0-b7d4-657f48389486","Type":"ContainerStarted","Data":"26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24"} Oct 30 01:07:23.541003 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.540696 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" event={"ID":"ef083d76-7f00-49e0-b7d4-657f48389486","Type":"ContainerStarted","Data":"908cd4a3efb7e3c61e7f5e0d38d15ce4cd09f90e7c2730696ac783a82fee0b9e"} Oct 30 01:07:23.611849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.609797 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" podStartSLOduration=3.609779808 podStartE2EDuration="3.609779808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:23.606006531 +0000 UTC m=+1982.177932754" watchObservedRunningTime="2026-10-30 01:07:23.609779808 +0000 UTC m=+1982.181706031" Oct 30 01:07:23.618517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.618484 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:23.920645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:23.920181 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.167586 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-certs\") pod \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.167667 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-podinfo\") pod \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.167704 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-envoy\") pod \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.167756 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-kube-api-access-jkrrn\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-kube-api-access-jkrrn\") pod \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.167831 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-socket\") pod \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.167929 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-data\") pod \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.168065 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istiod-ca-cert\") pod \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.168140 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-credential-socket\") pod \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.168303 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-token\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-token\") pod \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\" (UID: \"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a\") " Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.169076 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istiod-ca-cert" pod "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" (UID: "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.169325 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-data" pod "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" (UID: "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.169512 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-certs" pod "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" (UID: "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.169863 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-credential-socket" pod "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" (UID: "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:24.170606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.169925 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-socket" pod "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" (UID: "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:24.198596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.172044 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:24.198596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.172070 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:24.198596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.172084 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:24.198596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.172097 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:24.198596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.172122 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:24.248268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.248232 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-podinfo" pod "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" (UID: "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:24.250337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.250157 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-token" pod "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" (UID: "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:24.250337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.250275 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-envoy" pod "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" (UID: "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:24.252302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.252268 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-kube-api-access-jkrrn" pod "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" (UID: "2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a"). InnerVolumeSpecName "kube-api-access-jkrrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:24.297565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.277514 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:24.297565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.277548 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:24.297565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.277589 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:24.297565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.277603 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrrn\" (UniqueName: \"kubernetes.io/projected/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a-kube-api-access-jkrrn\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:07:24.473023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.472932 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:07:24.473293 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.473260 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 01:07:24.552169 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.552128 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" event={"ID":"2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a","Type":"ContainerDied","Data":"faab267407cf1d6acf3622f7d558f8c3395dff7d648edeafe7c784e8c1f5e3b0"} Oct 30 01:07:24.552764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.552203 4927 scope.go:122] "RemoveContainer" containerID="6d1b139e4634327f1f4ccf494c5f3182f858e448eb8ec7fff9e74576fc6513c8" Oct 30 01:07:24.552764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.552314 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv" Oct 30 01:07:24.572626 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.572594 4927 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:07:24.572738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.572646 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:07:24.646371 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.646342 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv"] Oct 30 01:07:24.653020 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:24.652990 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-h24nv"] Oct 30 01:07:25.473005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:25.472954 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:07:25.473183 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:25.473013 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 01:07:25.572669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:25.570137 4927 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:07:25.572669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:25.570193 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:07:25.629604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:25.628977 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" path="/var/lib/kubelet/pods/2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a/volumes" Oct 30 01:07:26.479048 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:26.479012 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:07:26.479357 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:26.479319 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 01:07:26.628013 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:26.627976 4927 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:07:26.628642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:26.628604 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:07:27.481924 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:27.481887 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:27.484337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:27.484314 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:27.486782 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:27.486760 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:07:27.570524 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:27.570488 4927 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:07:27.570760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:27.570553 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:07:28.570043 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:28.570009 4927 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:07:28.570660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:28.570629 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:07:29.570725 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:29.570686 4927 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:07:29.571236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:29.570766 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 01:07:30.575255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:30.575219 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:30.575797 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:30.575604 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:30.576430 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:07:30.576406 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:07:34.444522 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:07:34.444487 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:08:00.192176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:00.190717 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z"] Oct 30 01:08:00.192176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:00.191000 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" containerID="cri-o://26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24" gracePeriod=30 Oct 30 01:08:00.578979 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:00.578908 4927 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:08:00.578979 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:00.578970 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:08:04.444336 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:08:04.444251 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:08:05.838906 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.838824 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:08:05.854337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854306 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-credential-socket\") pod \"ef083d76-7f00-49e0-b7d4-657f48389486\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " Oct 30 01:08:05.854471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854438 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef083d76-7f00-49e0-b7d4-657f48389486-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef083d76-7f00-49e0-b7d4-657f48389486-istiod-ca-cert\") pod \"ef083d76-7f00-49e0-b7d4-657f48389486\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " Oct 30 01:08:05.854544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854518 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-data\") pod \"ef083d76-7f00-49e0-b7d4-657f48389486\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " Oct 30 01:08:05.854621 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854557 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-socket\") pod \"ef083d76-7f00-49e0-b7d4-657f48389486\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " Oct 30 01:08:05.854621 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854602 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-kube-api-access-ztqd5\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-kube-api-access-ztqd5\") pod \"ef083d76-7f00-49e0-b7d4-657f48389486\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " Oct 30 01:08:05.854695 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854631 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-credential-socket" pod "ef083d76-7f00-49e0-b7d4-657f48389486" (UID: "ef083d76-7f00-49e0-b7d4-657f48389486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.854695 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854646 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-certs\") pod \"ef083d76-7f00-49e0-b7d4-657f48389486\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " Oct 30 01:08:05.854804 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854776 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-istio-token\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-istio-token\") pod \"ef083d76-7f00-49e0-b7d4-657f48389486\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " Oct 30 01:08:05.854898 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854880 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef083d76-7f00-49e0-b7d4-657f48389486-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef083d76-7f00-49e0-b7d4-657f48389486-istio-podinfo\") pod \"ef083d76-7f00-49e0-b7d4-657f48389486\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " Oct 30 01:08:05.854898 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854882 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-data" pod "ef083d76-7f00-49e0-b7d4-657f48389486" (UID: "ef083d76-7f00-49e0-b7d4-657f48389486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.855012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854893 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-socket" pod "ef083d76-7f00-49e0-b7d4-657f48389486" (UID: "ef083d76-7f00-49e0-b7d4-657f48389486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.855012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.854917 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef083d76-7f00-49e0-b7d4-657f48389486-istiod-ca-cert" pod "ef083d76-7f00-49e0-b7d4-657f48389486" (UID: "ef083d76-7f00-49e0-b7d4-657f48389486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:05.855125 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.855047 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-certs" pod "ef083d76-7f00-49e0-b7d4-657f48389486" (UID: "ef083d76-7f00-49e0-b7d4-657f48389486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.855207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.855186 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-envoy\") pod \"ef083d76-7f00-49e0-b7d4-657f48389486\" (UID: \"ef083d76-7f00-49e0-b7d4-657f48389486\") " Oct 30 01:08:05.856055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.856013 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.856055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.856044 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef083d76-7f00-49e0-b7d4-657f48389486-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.856268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.856060 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.856268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.856073 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.856268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.856087 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.857486 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.857462 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-istio-token" pod "ef083d76-7f00-49e0-b7d4-657f48389486" (UID: "ef083d76-7f00-49e0-b7d4-657f48389486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:05.857619 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.857505 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef083d76-7f00-49e0-b7d4-657f48389486-istio-podinfo" pod "ef083d76-7f00-49e0-b7d4-657f48389486" (UID: "ef083d76-7f00-49e0-b7d4-657f48389486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:05.857725 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.857697 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-envoy" pod "ef083d76-7f00-49e0-b7d4-657f48389486" (UID: "ef083d76-7f00-49e0-b7d4-657f48389486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:05.859785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.859764 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-kube-api-access-ztqd5" pod "ef083d76-7f00-49e0-b7d4-657f48389486" (UID: "ef083d76-7f00-49e0-b7d4-657f48389486"). InnerVolumeSpecName "kube-api-access-ztqd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:05.957891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.957859 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqd5\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-kube-api-access-ztqd5\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.957891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.957891 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef083d76-7f00-49e0-b7d4-657f48389486-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.958133 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.957907 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef083d76-7f00-49e0-b7d4-657f48389486-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:05.958133 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:05.957919 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef083d76-7f00-49e0-b7d4-657f48389486-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:06.190877 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:06.190834 4927 generic.go:356] "Generic (PLEG): container finished" podID="ef083d76-7f00-49e0-b7d4-657f48389486" containerID="26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24" exitCode=0 Oct 30 01:08:06.191061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:06.190895 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" Oct 30 01:08:06.191061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:06.190941 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" event={"ID":"ef083d76-7f00-49e0-b7d4-657f48389486","Type":"ContainerDied","Data":"26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24"} Oct 30 01:08:06.191061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:06.191015 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z" event={"ID":"ef083d76-7f00-49e0-b7d4-657f48389486","Type":"ContainerDied","Data":"908cd4a3efb7e3c61e7f5e0d38d15ce4cd09f90e7c2730696ac783a82fee0b9e"} Oct 30 01:08:06.191061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:06.191044 4927 scope.go:122] "RemoveContainer" containerID="26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24" Oct 30 01:08:06.205693 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:06.205676 4927 scope.go:122] "RemoveContainer" containerID="26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24" Oct 30 01:08:06.205962 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:08:06.205942 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24\": container with ID starting with 26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24 not found: ID does not exist" containerID="26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24" Oct 30 01:08:06.206024 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:06.205967 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24"} err="failed to get container status \"26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24\": rpc error: code = NotFound desc = could not find container \"26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24\": container with ID starting with 26b3e509eea14fed3d364f7ab0cba54b7da8e2c7e26a062a4331bbb6d029ea24 not found: ID does not exist" Oct 30 01:08:06.245904 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:06.245874 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z"] Oct 30 01:08:06.251235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:06.251210 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-m7x6z"] Oct 30 01:08:07.574783 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:07.574741 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" path="/var/lib/kubelet/pods/ef083d76-7f00-49e0-b7d4-657f48389486/volumes" Oct 30 01:08:22.491622 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:22.491562 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m"] Oct 30 01:08:33.076917 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:33.076884 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx"] Oct 30 01:08:33.077494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:33.077111 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" podUID="1e0c2d70-f1da-49bb-8969-3c78aad572b1" containerName="maas-api" containerID="cri-o://0918793ad5038eacd7583891209cc090e0f73ffca75adf42a7689aa8281d59cf" gracePeriod=30 Oct 30 01:08:33.910858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:33.910824 4927 generic.go:356] "Generic (PLEG): container finished" podID="1e0c2d70-f1da-49bb-8969-3c78aad572b1" containerID="0918793ad5038eacd7583891209cc090e0f73ffca75adf42a7689aa8281d59cf" exitCode=0 Oct 30 01:08:33.915966 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:33.914162 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:08:33.918395 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:33.918368 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" event={"ID":"1e0c2d70-f1da-49bb-8969-3c78aad572b1","Type":"ContainerDied","Data":"0918793ad5038eacd7583891209cc090e0f73ffca75adf42a7689aa8281d59cf"} Oct 30 01:08:33.918537 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:33.918439 4927 scope.go:122] "RemoveContainer" containerID="0918793ad5038eacd7583891209cc090e0f73ffca75adf42a7689aa8281d59cf" Oct 30 01:08:34.079962 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.079929 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-metrics-certs\") pod \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " Oct 30 01:08:34.080410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.080070 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls\") pod \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " Oct 30 01:08:34.080410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.080195 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0c2d70-f1da-49bb-8969-3c78aad572b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0c2d70-f1da-49bb-8969-3c78aad572b1-openshift-service-ca\") pod \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " Oct 30 01:08:34.080410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.080269 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0c2d70-f1da-49bb-8969-3c78aad572b1-kube-api-access-vpwlv\" (UniqueName: \"kubernetes.io/projected/1e0c2d70-f1da-49bb-8969-3c78aad572b1-kube-api-access-vpwlv\") pod \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\" (UID: \"1e0c2d70-f1da-49bb-8969-3c78aad572b1\") " Oct 30 01:08:34.080717 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.080687 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0c2d70-f1da-49bb-8969-3c78aad572b1-openshift-service-ca" pod "1e0c2d70-f1da-49bb-8969-3c78aad572b1" (UID: "1e0c2d70-f1da-49bb-8969-3c78aad572b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:34.081266 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.081237 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0c2d70-f1da-49bb-8969-3c78aad572b1-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:34.082245 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.082213 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-metrics-certs" pod "1e0c2d70-f1da-49bb-8969-3c78aad572b1" (UID: "1e0c2d70-f1da-49bb-8969-3c78aad572b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:34.082354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.082250 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls" pod "1e0c2d70-f1da-49bb-8969-3c78aad572b1" (UID: "1e0c2d70-f1da-49bb-8969-3c78aad572b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:34.082452 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.082432 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0c2d70-f1da-49bb-8969-3c78aad572b1-kube-api-access-vpwlv" pod "1e0c2d70-f1da-49bb-8969-3c78aad572b1" (UID: "1e0c2d70-f1da-49bb-8969-3c78aad572b1"). InnerVolumeSpecName "kube-api-access-vpwlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:34.182632 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.182517 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:34.182632 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.182543 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0c2d70-f1da-49bb-8969-3c78aad572b1-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:34.182632 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.182553 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpwlv\" (UniqueName: \"kubernetes.io/projected/1e0c2d70-f1da-49bb-8969-3c78aad572b1-kube-api-access-vpwlv\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:34.445604 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:08:34.445500 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:08:34.923479 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.923450 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" Oct 30 01:08:34.923672 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.923494 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx" event={"ID":"1e0c2d70-f1da-49bb-8969-3c78aad572b1","Type":"ContainerDied","Data":"6c2936c8237aae15779040649aead00ec7dbe031ade80cf52c6281b44820c51b"} Oct 30 01:08:34.978803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.978773 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx"] Oct 30 01:08:34.988303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:34.988273 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-820c04-55fdfcc7f4-qknzx"] Oct 30 01:08:35.574507 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:35.574468 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0c2d70-f1da-49bb-8969-3c78aad572b1" path="/var/lib/kubelet/pods/1e0c2d70-f1da-49bb-8969-3c78aad572b1/volumes" Oct 30 01:08:37.514959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.514919 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" podUID="faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" containerName="main" containerID="cri-o://09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f" gracePeriod=45 Oct 30 01:08:37.883690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.883664 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:08:37.953541 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.953505 4927 generic.go:356] "Generic (PLEG): container finished" podID="faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" containerID="09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f" exitCode=0 Oct 30 01:08:37.953744 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.953734 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" event={"ID":"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8","Type":"ContainerDied","Data":"09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f"} Oct 30 01:08:37.953789 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.953763 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" event={"ID":"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8","Type":"ContainerDied","Data":"f7540737cf2b8e43af36306aa9883ff5b7e61c09bc6527a1b8e432737441849f"} Oct 30 01:08:37.953789 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.953781 4927 scope.go:122] "RemoveContainer" containerID="09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f" Oct 30 01:08:37.953882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.953871 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" Oct 30 01:08:37.966790 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.966764 4927 scope.go:122] "RemoveContainer" containerID="09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f" Oct 30 01:08:37.967346 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:08:37.967310 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f\": container with ID starting with 09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f not found: ID does not exist" containerID="09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f" Oct 30 01:08:37.967458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.967359 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f"} err="failed to get container status \"09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f\": rpc error: code = NotFound desc = could not find container \"09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f\": container with ID starting with 09804d5a51d422839a98f2e289d09f687650daaaedbc287e905a2e33b336e45f not found: ID does not exist" Oct 30 01:08:37.984249 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.984224 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-dshm\") pod \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " Oct 30 01:08:37.984372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.984262 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tls-certs\") pod \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " Oct 30 01:08:37.984372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.984308 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-home\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-home\") pod \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " Oct 30 01:08:37.984713 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.984479 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-model-cache\") pod \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " Oct 30 01:08:37.984713 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.984629 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-home" pod "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" (UID: "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:37.984845 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.984776 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-model-cache" pod "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" (UID: "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:37.984953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.984936 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tmp-dir\") pod \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " Oct 30 01:08:37.985086 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.985071 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-kube-api-access-krmph\" (UniqueName: \"kubernetes.io/projected/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-kube-api-access-krmph\") pod \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\" (UID: \"faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8\") " Oct 30 01:08:37.985208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.985149 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tmp-dir" pod "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" (UID: "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:37.985830 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.985811 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:37.985916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.985833 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:37.985916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.985845 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:37.987235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.987204 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-kube-api-access-krmph" pod "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" (UID: "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8"). InnerVolumeSpecName "kube-api-access-krmph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:37.987326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.987244 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tls-certs" pod "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" (UID: "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:37.987326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:37.987253 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-dshm" pod "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" (UID: "faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:38.087974 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:38.087903 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krmph\" (UniqueName: \"kubernetes.io/projected/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-kube-api-access-krmph\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:38.087974 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:38.087930 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:38.087974 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:38.087941 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:38.305620 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:38.304505 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m"] Oct 30 01:08:38.321898 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:38.321865 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m"] Oct 30 01:08:38.857508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:38.857462 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49dfaf/test-model-49dfaf-kserve-88949d747-p5f6m" podUID="faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:08:39.583192 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:39.583148 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" path="/var/lib/kubelet/pods/faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8/volumes" Oct 30 01:08:42.530292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:42.530255 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh"] Oct 30 01:08:51.797940 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:51.797908 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm"] Oct 30 01:08:51.831766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:51.831733 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9"] Oct 30 01:08:51.832218 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:51.832190 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" containerID="cri-o://191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06" gracePeriod=30 Oct 30 01:08:53.632939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:53.632904 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56"] Oct 30 01:08:57.120697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.120625 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 01:08:57.218767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.218733 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-token\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-token\") pod \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " Oct 30 01:08:57.218933 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.218897 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-envoy\") pod \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " Oct 30 01:08:57.218991 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.218947 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-kube-api-access-97qzc\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-kube-api-access-97qzc\") pod \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " Oct 30 01:08:57.219053 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219026 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-credential-socket\") pod \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " Oct 30 01:08:57.219108 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219073 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istiod-ca-cert\") pod \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " Oct 30 01:08:57.219171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219106 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-podinfo\") pod \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " Oct 30 01:08:57.219171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219159 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-socket\") pod \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " Oct 30 01:08:57.219333 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219311 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-certs\") pod \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " Oct 30 01:08:57.219552 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219533 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-data\") pod \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\" (UID: \"4e5b9fe3-ac8b-4727-99ac-47a588050de8\") " Oct 30 01:08:57.219697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219383 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-credential-socket" pod "4e5b9fe3-ac8b-4727-99ac-47a588050de8" (UID: "4e5b9fe3-ac8b-4727-99ac-47a588050de8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:57.219781 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219664 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-socket" pod "4e5b9fe3-ac8b-4727-99ac-47a588050de8" (UID: "4e5b9fe3-ac8b-4727-99ac-47a588050de8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:57.219984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219961 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-data" pod "4e5b9fe3-ac8b-4727-99ac-47a588050de8" (UID: "4e5b9fe3-ac8b-4727-99ac-47a588050de8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:57.220061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.219978 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-certs" pod "4e5b9fe3-ac8b-4727-99ac-47a588050de8" (UID: "4e5b9fe3-ac8b-4727-99ac-47a588050de8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:57.220212 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.220184 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istiod-ca-cert" pod "4e5b9fe3-ac8b-4727-99ac-47a588050de8" (UID: "4e5b9fe3-ac8b-4727-99ac-47a588050de8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:57.220919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.220893 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.221026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.220924 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.221026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.220941 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.221026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.220954 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.221026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.220968 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.221827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.221624 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-kube-api-access-97qzc" pod "4e5b9fe3-ac8b-4727-99ac-47a588050de8" (UID: "4e5b9fe3-ac8b-4727-99ac-47a588050de8"). InnerVolumeSpecName "kube-api-access-97qzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:57.221827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.221711 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-token" pod "4e5b9fe3-ac8b-4727-99ac-47a588050de8" (UID: "4e5b9fe3-ac8b-4727-99ac-47a588050de8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:57.221827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.221710 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-envoy" pod "4e5b9fe3-ac8b-4727-99ac-47a588050de8" (UID: "4e5b9fe3-ac8b-4727-99ac-47a588050de8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:57.222245 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.222216 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-podinfo" pod "4e5b9fe3-ac8b-4727-99ac-47a588050de8" (UID: "4e5b9fe3-ac8b-4727-99ac-47a588050de8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:57.295728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.295685 4927 generic.go:356] "Generic (PLEG): container finished" podID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerID="191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06" exitCode=0 Oct 30 01:08:57.295918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.295734 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" Oct 30 01:08:57.295918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.295764 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" event={"ID":"4e5b9fe3-ac8b-4727-99ac-47a588050de8","Type":"ContainerDied","Data":"191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06"} Oct 30 01:08:57.295918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.295828 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9" event={"ID":"4e5b9fe3-ac8b-4727-99ac-47a588050de8","Type":"ContainerDied","Data":"169b380ceb23d62920a96d928707376a9872183e15fc63177efee1b1efe6a459"} Oct 30 01:08:57.295918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.295859 4927 scope.go:122] "RemoveContainer" containerID="191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06" Oct 30 01:08:57.310500 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.310478 4927 scope.go:122] "RemoveContainer" containerID="191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06" Oct 30 01:08:57.310828 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:08:57.310807 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06\": container with ID starting with 191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06 not found: ID does not exist" containerID="191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06" Oct 30 01:08:57.310913 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.310841 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06"} err="failed to get container status \"191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06\": rpc error: code = NotFound desc = could not find container \"191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06\": container with ID starting with 191565fefe09615ab39f529128d8c405fac2e17e2a4e8de19cc129efc538ec06 not found: ID does not exist" Oct 30 01:08:57.322931 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.322908 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.323037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.322933 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97qzc\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-kube-api-access-97qzc\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.323037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.322944 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.323037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.322952 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5b9fe3-ac8b-4727-99ac-47a588050de8-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.359834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.359800 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9"] Oct 30 01:08:57.367306 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.367281 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-z7sb9"] Oct 30 01:08:57.559443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.559383 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" podUID="6a54e8b1-b885-49c6-b803-5cad96e73314" containerName="main" containerID="cri-o://732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940" gracePeriod=45 Oct 30 01:08:57.574481 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.574455 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" path="/var/lib/kubelet/pods/4e5b9fe3-ac8b-4727-99ac-47a588050de8/volumes" Oct 30 01:08:57.833779 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.833749 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:08:57.932452 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.932421 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-home\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-home\") pod \"6a54e8b1-b885-49c6-b803-5cad96e73314\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " Oct 30 01:08:57.932702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.932684 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-dshm\") pod \"6a54e8b1-b885-49c6-b803-5cad96e73314\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " Oct 30 01:08:57.932803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.932729 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-model-cache\") pod \"6a54e8b1-b885-49c6-b803-5cad96e73314\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " Oct 30 01:08:57.932803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.932742 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-home" pod "6a54e8b1-b885-49c6-b803-5cad96e73314" (UID: "6a54e8b1-b885-49c6-b803-5cad96e73314"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:57.932956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.932941 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a54e8b1-b885-49c6-b803-5cad96e73314-kube-api-access-vhsf6\" (UniqueName: \"kubernetes.io/projected/6a54e8b1-b885-49c6-b803-5cad96e73314-kube-api-access-vhsf6\") pod \"6a54e8b1-b885-49c6-b803-5cad96e73314\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " Oct 30 01:08:57.933094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.933054 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-model-cache" pod "6a54e8b1-b885-49c6-b803-5cad96e73314" (UID: "6a54e8b1-b885-49c6-b803-5cad96e73314"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:57.933426 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.933404 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a54e8b1-b885-49c6-b803-5cad96e73314-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a54e8b1-b885-49c6-b803-5cad96e73314-tls-certs\") pod \"6a54e8b1-b885-49c6-b803-5cad96e73314\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " Oct 30 01:08:57.933502 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.933487 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-tmp-dir\") pod \"6a54e8b1-b885-49c6-b803-5cad96e73314\" (UID: \"6a54e8b1-b885-49c6-b803-5cad96e73314\") " Oct 30 01:08:57.934032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.934003 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-tmp-dir" pod "6a54e8b1-b885-49c6-b803-5cad96e73314" (UID: "6a54e8b1-b885-49c6-b803-5cad96e73314"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:57.934477 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.934454 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.934564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.934481 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.934564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.934494 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:57.935439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.935412 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-dshm" pod "6a54e8b1-b885-49c6-b803-5cad96e73314" (UID: "6a54e8b1-b885-49c6-b803-5cad96e73314"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:57.935554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.935441 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a54e8b1-b885-49c6-b803-5cad96e73314-kube-api-access-vhsf6" pod "6a54e8b1-b885-49c6-b803-5cad96e73314" (UID: "6a54e8b1-b885-49c6-b803-5cad96e73314"). InnerVolumeSpecName "kube-api-access-vhsf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:57.936538 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:57.936519 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a54e8b1-b885-49c6-b803-5cad96e73314-tls-certs" pod "6a54e8b1-b885-49c6-b803-5cad96e73314" (UID: "6a54e8b1-b885-49c6-b803-5cad96e73314"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:58.035803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.035769 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhsf6\" (UniqueName: \"kubernetes.io/projected/6a54e8b1-b885-49c6-b803-5cad96e73314-kube-api-access-vhsf6\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:58.035803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.035796 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a54e8b1-b885-49c6-b803-5cad96e73314-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:58.035803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.035806 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a54e8b1-b885-49c6-b803-5cad96e73314-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:58.308222 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.308190 4927 generic.go:356] "Generic (PLEG): container finished" podID="6a54e8b1-b885-49c6-b803-5cad96e73314" containerID="732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940" exitCode=0 Oct 30 01:08:58.308729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.308239 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" Oct 30 01:08:58.308729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.308263 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" event={"ID":"6a54e8b1-b885-49c6-b803-5cad96e73314","Type":"ContainerDied","Data":"732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940"} Oct 30 01:08:58.308729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.308300 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh" event={"ID":"6a54e8b1-b885-49c6-b803-5cad96e73314","Type":"ContainerDied","Data":"a021ba06aec747d95509f15b81faa2b891374b206956e13fe21096e39c588c00"} Oct 30 01:08:58.308729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.308320 4927 scope.go:122] "RemoveContainer" containerID="732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940" Oct 30 01:08:58.321057 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.321035 4927 scope.go:122] "RemoveContainer" containerID="732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940" Oct 30 01:08:58.321319 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:08:58.321299 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940\": container with ID starting with 732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940 not found: ID does not exist" containerID="732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940" Oct 30 01:08:58.321396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.321327 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940"} err="failed to get container status \"732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940\": rpc error: code = NotFound desc = could not find container \"732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940\": container with ID starting with 732fcfbdcafb0691f68debb5a2e4a679b3c56c3bc00f448b2ffbbf47890e7940 not found: ID does not exist" Oct 30 01:08:58.362023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.361995 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh"] Oct 30 01:08:58.373548 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.373522 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-97ceef/test-model-97ceef-kserve-d4bff645c-prddh"] Oct 30 01:08:58.635697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.635636 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" podUID="3ffc62a3-2f1e-475d-9073-2ecab906fe07" containerName="payload-processing" containerID="cri-o://520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3" gracePeriod=25 Oct 30 01:08:58.950129 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:58.950104 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:08:59.053079 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.053037 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ffc62a3-2f1e-475d-9073-2ecab906fe07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffc62a3-2f1e-475d-9073-2ecab906fe07-plugins-config-volume\") pod \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\" (UID: \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\") " Oct 30 01:08:59.053278 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.053146 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffc62a3-2f1e-475d-9073-2ecab906fe07-kube-api-access-8ms5w\" (UniqueName: \"kubernetes.io/projected/3ffc62a3-2f1e-475d-9073-2ecab906fe07-kube-api-access-8ms5w\") pod \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\" (UID: \"3ffc62a3-2f1e-475d-9073-2ecab906fe07\") " Oct 30 01:08:59.053545 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.053495 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffc62a3-2f1e-475d-9073-2ecab906fe07-plugins-config-volume" pod "3ffc62a3-2f1e-475d-9073-2ecab906fe07" (UID: "3ffc62a3-2f1e-475d-9073-2ecab906fe07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:59.054118 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.054093 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffc62a3-2f1e-475d-9073-2ecab906fe07-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:59.055392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.055352 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffc62a3-2f1e-475d-9073-2ecab906fe07-kube-api-access-8ms5w" pod "3ffc62a3-2f1e-475d-9073-2ecab906fe07" (UID: "3ffc62a3-2f1e-475d-9073-2ecab906fe07"). InnerVolumeSpecName "kube-api-access-8ms5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:59.156416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.156333 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ms5w\" (UniqueName: \"kubernetes.io/projected/3ffc62a3-2f1e-475d-9073-2ecab906fe07-kube-api-access-8ms5w\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:08:59.318007 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.317975 4927 generic.go:356] "Generic (PLEG): container finished" podID="3ffc62a3-2f1e-475d-9073-2ecab906fe07" containerID="520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3" exitCode=0 Oct 30 01:08:59.318490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.318023 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" Oct 30 01:08:59.318490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.318060 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" event={"ID":"3ffc62a3-2f1e-475d-9073-2ecab906fe07","Type":"ContainerDied","Data":"520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3"} Oct 30 01:08:59.318490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.318141 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56" event={"ID":"3ffc62a3-2f1e-475d-9073-2ecab906fe07","Type":"ContainerDied","Data":"9b618e8524c46484c7dbce1c7a663dcab69550cbc504d62284cf6e7467f28dcd"} Oct 30 01:08:59.318490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.318164 4927 scope.go:122] "RemoveContainer" containerID="520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3" Oct 30 01:08:59.330006 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.329986 4927 scope.go:122] "RemoveContainer" containerID="520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3" Oct 30 01:08:59.330291 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:08:59.330271 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3\": container with ID starting with 520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3 not found: ID does not exist" containerID="520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3" Oct 30 01:08:59.330379 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.330297 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3"} err="failed to get container status \"520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3\": rpc error: code = NotFound desc = could not find container \"520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3\": container with ID starting with 520ea15adbb990b01596270d07c3d72ae7b5835dca4a6f56ff18c782d07bb6b3 not found: ID does not exist" Oct 30 01:08:59.369428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.369393 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56"] Oct 30 01:08:59.380210 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.380183 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49dfaf-b8746c684-mpl56"] Oct 30 01:08:59.574275 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.574190 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ffc62a3-2f1e-475d-9073-2ecab906fe07" path="/var/lib/kubelet/pods/3ffc62a3-2f1e-475d-9073-2ecab906fe07/volumes" Oct 30 01:08:59.574851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:08:59.574827 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a54e8b1-b885-49c6-b803-5cad96e73314" path="/var/lib/kubelet/pods/6a54e8b1-b885-49c6-b803-5cad96e73314/volumes" Oct 30 01:09:04.447022 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:04.446980 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:09:06.823480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:06.823426 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" podUID="365d784b-a986-49a5-ba10-93a80bdf2630" containerName="main" containerID="cri-o://99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef" gracePeriod=45 Oct 30 01:09:07.138076 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.138042 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:09:07.239596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.238626 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365d784b-a986-49a5-ba10-93a80bdf2630-tls-certs\" (UniqueName: \"kubernetes.io/secret/365d784b-a986-49a5-ba10-93a80bdf2630-tls-certs\") pod \"365d784b-a986-49a5-ba10-93a80bdf2630\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " Oct 30 01:09:07.241662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.240156 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-tmp-dir" pod "365d784b-a986-49a5-ba10-93a80bdf2630" (UID: "365d784b-a986-49a5-ba10-93a80bdf2630"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:07.241662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.239633 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-tmp-dir\") pod \"365d784b-a986-49a5-ba10-93a80bdf2630\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " Oct 30 01:09:07.241662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.240385 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-model-cache\") pod \"365d784b-a986-49a5-ba10-93a80bdf2630\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " Oct 30 01:09:07.241662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.241486 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-model-cache" pod "365d784b-a986-49a5-ba10-93a80bdf2630" (UID: "365d784b-a986-49a5-ba10-93a80bdf2630"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:07.241988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.241708 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-home\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-home\") pod \"365d784b-a986-49a5-ba10-93a80bdf2630\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " Oct 30 01:09:07.244677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.242066 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-home" pod "365d784b-a986-49a5-ba10-93a80bdf2630" (UID: "365d784b-a986-49a5-ba10-93a80bdf2630"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:07.244677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.242388 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-dshm\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-dshm\") pod \"365d784b-a986-49a5-ba10-93a80bdf2630\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " Oct 30 01:09:07.297844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.297796 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-dshm" pod "365d784b-a986-49a5-ba10-93a80bdf2630" (UID: "365d784b-a986-49a5-ba10-93a80bdf2630"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:07.297844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.297856 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365d784b-a986-49a5-ba10-93a80bdf2630-kube-api-access-hlzhd\" (UniqueName: \"kubernetes.io/projected/365d784b-a986-49a5-ba10-93a80bdf2630-kube-api-access-hlzhd\") pod \"365d784b-a986-49a5-ba10-93a80bdf2630\" (UID: \"365d784b-a986-49a5-ba10-93a80bdf2630\") " Oct 30 01:09:07.300928 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.298757 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:07.300928 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.298778 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:07.300928 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.298791 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:07.300928 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.298804 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/365d784b-a986-49a5-ba10-93a80bdf2630-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:07.301678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.301544 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365d784b-a986-49a5-ba10-93a80bdf2630-kube-api-access-hlzhd" pod "365d784b-a986-49a5-ba10-93a80bdf2630" (UID: "365d784b-a986-49a5-ba10-93a80bdf2630"). InnerVolumeSpecName "kube-api-access-hlzhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:07.312879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.312852 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365d784b-a986-49a5-ba10-93a80bdf2630-tls-certs" pod "365d784b-a986-49a5-ba10-93a80bdf2630" (UID: "365d784b-a986-49a5-ba10-93a80bdf2630"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:07.400042 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.400014 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlzhd\" (UniqueName: \"kubernetes.io/projected/365d784b-a986-49a5-ba10-93a80bdf2630-kube-api-access-hlzhd\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:07.400042 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.400040 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/365d784b-a986-49a5-ba10-93a80bdf2630-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:07.440510 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.440480 4927 generic.go:356] "Generic (PLEG): container finished" podID="365d784b-a986-49a5-ba10-93a80bdf2630" containerID="99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef" exitCode=0 Oct 30 01:09:07.440729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.440533 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" Oct 30 01:09:07.440729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.440566 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" event={"ID":"365d784b-a986-49a5-ba10-93a80bdf2630","Type":"ContainerDied","Data":"99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef"} Oct 30 01:09:07.440729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.440632 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" event={"ID":"365d784b-a986-49a5-ba10-93a80bdf2630","Type":"ContainerDied","Data":"2b4e7801bc0ea8944e7043dce9ef56e23466c4436a7abf1f038dac9a1803c3eb"} Oct 30 01:09:07.440729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.440656 4927 scope.go:122] "RemoveContainer" containerID="99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef" Oct 30 01:09:07.452669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.452652 4927 scope.go:122] "RemoveContainer" containerID="99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef" Oct 30 01:09:07.452924 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:07.452903 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef\": container with ID starting with 99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef not found: ID does not exist" containerID="99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef" Oct 30 01:09:07.452987 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.452934 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef"} err="failed to get container status \"99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef\": rpc error: code = NotFound desc = could not find container \"99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef\": container with ID starting with 99c1d474b0ef8850c4659b2abb24063f1dcb05f179e6ed0e181b133f4f78e9ef not found: ID does not exist" Oct 30 01:09:07.492166 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.492136 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm"] Oct 30 01:09:07.503026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.503002 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm"] Oct 30 01:09:07.583463 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:07.583417 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365d784b-a986-49a5-ba10-93a80bdf2630" path="/var/lib/kubelet/pods/365d784b-a986-49a5-ba10-93a80bdf2630/volumes" Oct 30 01:09:08.058143 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:08.058103 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1da7b2/test-model-1da7b2-kserve-74549b69d9-gjdtm" podUID="365d784b-a986-49a5-ba10-93a80bdf2630" containerName="main" probeResult="failure" output="Get \"https://10.132.0.158:8000/ready\": context deadline exceeded" Oct 30 01:09:19.039625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.039585 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv"] Oct 30 01:09:19.040205 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.039843 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" podUID="92fe5a50-15bf-4ff7-ab5b-9540ce501400" containerName="maas-api" containerID="cri-o://45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d" gracePeriod=30 Oct 30 01:09:19.749591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.749015 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:09:19.770655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.767720 4927 generic.go:356] "Generic (PLEG): container finished" podID="92fe5a50-15bf-4ff7-ab5b-9540ce501400" containerID="45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d" exitCode=0 Oct 30 01:09:19.770655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.767792 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" Oct 30 01:09:19.770655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.767800 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" event={"ID":"92fe5a50-15bf-4ff7-ab5b-9540ce501400","Type":"ContainerDied","Data":"45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d"} Oct 30 01:09:19.770655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.767867 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv" event={"ID":"92fe5a50-15bf-4ff7-ab5b-9540ce501400","Type":"ContainerDied","Data":"7e1b1719c984ba4a02c5b53236151863b6d20c02bab52cec88fcf87abca49766"} Oct 30 01:09:19.770655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.767896 4927 scope.go:122] "RemoveContainer" containerID="45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d" Oct 30 01:09:19.781673 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.781644 4927 scope.go:122] "RemoveContainer" containerID="45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d" Oct 30 01:09:19.782023 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:19.782002 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d\": container with ID starting with 45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d not found: ID does not exist" containerID="45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d" Oct 30 01:09:19.782098 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.782031 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d"} err="failed to get container status \"45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d\": rpc error: code = NotFound desc = could not find container \"45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d\": container with ID starting with 45f37d9b83dadf0605da5c62ad5813f5bec8ed372e477e21897099144925116d not found: ID does not exist" Oct 30 01:09:19.853198 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.853169 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-maas-api-tls\") pod \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " Oct 30 01:09:19.853399 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.853352 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fe5a50-15bf-4ff7-ab5b-9540ce501400-kube-api-access-drjbq\" (UniqueName: \"kubernetes.io/projected/92fe5a50-15bf-4ff7-ab5b-9540ce501400-kube-api-access-drjbq\") pod \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " Oct 30 01:09:19.853548 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.853530 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-metrics-certs\") pod \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " Oct 30 01:09:19.853938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.853664 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92fe5a50-15bf-4ff7-ab5b-9540ce501400-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe5a50-15bf-4ff7-ab5b-9540ce501400-openshift-service-ca\") pod \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\" (UID: \"92fe5a50-15bf-4ff7-ab5b-9540ce501400\") " Oct 30 01:09:19.855637 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.854428 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fe5a50-15bf-4ff7-ab5b-9540ce501400-openshift-service-ca" pod "92fe5a50-15bf-4ff7-ab5b-9540ce501400" (UID: "92fe5a50-15bf-4ff7-ab5b-9540ce501400"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:19.858059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.858023 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-metrics-certs" pod "92fe5a50-15bf-4ff7-ab5b-9540ce501400" (UID: "92fe5a50-15bf-4ff7-ab5b-9540ce501400"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:19.858239 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.858209 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fe5a50-15bf-4ff7-ab5b-9540ce501400-kube-api-access-drjbq" pod "92fe5a50-15bf-4ff7-ab5b-9540ce501400" (UID: "92fe5a50-15bf-4ff7-ab5b-9540ce501400"). InnerVolumeSpecName "kube-api-access-drjbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:19.860343 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.860322 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-maas-api-tls" pod "92fe5a50-15bf-4ff7-ab5b-9540ce501400" (UID: "92fe5a50-15bf-4ff7-ab5b-9540ce501400"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:19.955999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.955963 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drjbq\" (UniqueName: \"kubernetes.io/projected/92fe5a50-15bf-4ff7-ab5b-9540ce501400-kube-api-access-drjbq\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:19.955999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.955994 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:19.955999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.956004 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe5a50-15bf-4ff7-ab5b-9540ce501400-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:19.956313 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:19.956013 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92fe5a50-15bf-4ff7-ab5b-9540ce501400-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:20.161958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:20.161926 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv"] Oct 30 01:09:20.179265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:20.179237 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49dfaf-7b769d9bc-qktpv"] Oct 30 01:09:21.576071 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:21.576036 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92fe5a50-15bf-4ff7-ab5b-9540ce501400" path="/var/lib/kubelet/pods/92fe5a50-15bf-4ff7-ab5b-9540ce501400/volumes" Oct 30 01:09:22.345554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:22.345522 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:09:22.354356 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:22.354323 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:09:25.041358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:25.041318 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw"] Oct 30 01:09:25.468653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:25.468586 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm"] Oct 30 01:09:27.241301 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:27.241263 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6"] Oct 30 01:09:27.241835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:27.241545 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" containerID="cri-o://ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9" gracePeriod=30 Oct 30 01:09:29.371642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:29.371605 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz"] Oct 30 01:09:29.940716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:29.940665 4927 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:09:29.940887 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:29.940758 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:09:30.142542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:30.142507 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr"] Oct 30 01:09:30.143100 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:30.143049 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" containerID="cri-o://9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84" gracePeriod=30 Oct 30 01:09:32.578750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.578722 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:09:32.632690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.632608 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5045c19-6086-4a68-801c-e3f7b3c098a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5045c19-6086-4a68-801c-e3f7b3c098a3-istiod-ca-cert\") pod \"d5045c19-6086-4a68-801c-e3f7b3c098a3\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " Oct 30 01:09:32.632878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.632758 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-socket\") pod \"d5045c19-6086-4a68-801c-e3f7b3c098a3\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " Oct 30 01:09:32.632878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.632830 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-data\") pod \"d5045c19-6086-4a68-801c-e3f7b3c098a3\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " Oct 30 01:09:32.632878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.632877 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-podinfo\") pod \"d5045c19-6086-4a68-801c-e3f7b3c098a3\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " Oct 30 01:09:32.633094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.632960 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5045c19-6086-4a68-801c-e3f7b3c098a3-istiod-ca-cert" pod "d5045c19-6086-4a68-801c-e3f7b3c098a3" (UID: "d5045c19-6086-4a68-801c-e3f7b3c098a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:32.633094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.632967 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-kube-api-access-fg5tf\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-kube-api-access-fg5tf\") pod \"d5045c19-6086-4a68-801c-e3f7b3c098a3\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " Oct 30 01:09:32.633271 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.633161 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-token\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-token\") pod \"d5045c19-6086-4a68-801c-e3f7b3c098a3\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " Oct 30 01:09:32.633271 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.633210 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-credential-socket\") pod \"d5045c19-6086-4a68-801c-e3f7b3c098a3\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " Oct 30 01:09:32.633271 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.633259 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-data" pod "d5045c19-6086-4a68-801c-e3f7b3c098a3" (UID: "d5045c19-6086-4a68-801c-e3f7b3c098a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:32.633443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.633359 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-certs\") pod \"d5045c19-6086-4a68-801c-e3f7b3c098a3\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " Oct 30 01:09:32.633443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.633397 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-envoy\") pod \"d5045c19-6086-4a68-801c-e3f7b3c098a3\" (UID: \"d5045c19-6086-4a68-801c-e3f7b3c098a3\") " Oct 30 01:09:32.634071 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.634003 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-socket" pod "d5045c19-6086-4a68-801c-e3f7b3c098a3" (UID: "d5045c19-6086-4a68-801c-e3f7b3c098a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:32.634704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.634356 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-certs" pod "d5045c19-6086-4a68-801c-e3f7b3c098a3" (UID: "d5045c19-6086-4a68-801c-e3f7b3c098a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:32.634984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.634842 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-credential-socket" pod "d5045c19-6086-4a68-801c-e3f7b3c098a3" (UID: "d5045c19-6086-4a68-801c-e3f7b3c098a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:32.635281 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.635253 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5045c19-6086-4a68-801c-e3f7b3c098a3-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:32.635386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.635289 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:32.635386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.635304 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:32.635386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.635320 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:32.635386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.635344 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:32.636456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.636427 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-kube-api-access-fg5tf" pod "d5045c19-6086-4a68-801c-e3f7b3c098a3" (UID: "d5045c19-6086-4a68-801c-e3f7b3c098a3"). InnerVolumeSpecName "kube-api-access-fg5tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:32.640613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.639950 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-podinfo" pod "d5045c19-6086-4a68-801c-e3f7b3c098a3" (UID: "d5045c19-6086-4a68-801c-e3f7b3c098a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:32.640613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.639985 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-token" pod "d5045c19-6086-4a68-801c-e3f7b3c098a3" (UID: "d5045c19-6086-4a68-801c-e3f7b3c098a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:32.640613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.640141 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-envoy" pod "d5045c19-6086-4a68-801c-e3f7b3c098a3" (UID: "d5045c19-6086-4a68-801c-e3f7b3c098a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:32.736517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.736480 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg5tf\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-kube-api-access-fg5tf\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:32.736517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.736514 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:32.736768 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.736527 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:32.736768 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.736544 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5045c19-6086-4a68-801c-e3f7b3c098a3-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:32.904606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.904560 4927 generic.go:356] "Generic (PLEG): container finished" podID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerID="ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9" exitCode=0 Oct 30 01:09:32.904779 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.904621 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" Oct 30 01:09:32.904779 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.904651 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" event={"ID":"d5045c19-6086-4a68-801c-e3f7b3c098a3","Type":"ContainerDied","Data":"ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9"} Oct 30 01:09:32.904779 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.904703 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6" event={"ID":"d5045c19-6086-4a68-801c-e3f7b3c098a3","Type":"ContainerDied","Data":"ef5ce404eaaf1cd0f84c09552489b708e61e3d556f69b46b873d1a725dba71a1"} Oct 30 01:09:32.904779 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.904726 4927 scope.go:122] "RemoveContainer" containerID="ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9" Oct 30 01:09:32.917649 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.917633 4927 scope.go:122] "RemoveContainer" containerID="e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30" Oct 30 01:09:32.928076 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.928057 4927 scope.go:122] "RemoveContainer" containerID="ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9" Oct 30 01:09:32.928408 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:32.928387 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9\": container with ID starting with ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9 not found: ID does not exist" containerID="ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9" Oct 30 01:09:32.928494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.928418 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9"} err="failed to get container status \"ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9\": rpc error: code = NotFound desc = could not find container \"ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9\": container with ID starting with ddac6269aa43e5ec82d380147a1bc1c0f3e7e13830a536cc66f4b4872e372fc9 not found: ID does not exist" Oct 30 01:09:32.928494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.928438 4927 scope.go:122] "RemoveContainer" containerID="e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30" Oct 30 01:09:32.928728 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:32.928708 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30\": container with ID starting with e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30 not found: ID does not exist" containerID="e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30" Oct 30 01:09:32.928772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.928737 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30"} err="failed to get container status \"e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30\": rpc error: code = NotFound desc = could not find container \"e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30\": container with ID starting with e1cd409eae97b43796fa21f307735fd7d193ee75b66110e1dc0a9573efc95a30 not found: ID does not exist" Oct 30 01:09:32.956094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.956072 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6"] Oct 30 01:09:32.969318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:32.969293 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-qlbg6"] Oct 30 01:09:33.573669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:33.573636 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" path="/var/lib/kubelet/pods/d5045c19-6086-4a68-801c-e3f7b3c098a3/volumes" Oct 30 01:09:33.622069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:33.622033 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6"] Oct 30 01:09:33.651367 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:33.651309 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-serving-cert: secret "maas-api-e2e-inf-a-780077-serving-cert" not found Oct 30 01:09:33.651556 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:33.651417 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls podName:a1a1e09b-20ef-4a61-833e-48e94dd4bbc6 nodeName:}" failed. No retries permitted until 2026-10-30 01:09:34.151393001 +0000 UTC m=+2112.723319225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls") pod "maas-api-e2e-inf-a-780077-7f49667468-6znm6" (UID: "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6") : secret "maas-api-e2e-inf-a-780077-serving-cert" not found Oct 30 01:09:33.964596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:33.961828 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" podUID="a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" containerName="maas-api" containerID="cri-o://3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293" gracePeriod=30 Oct 30 01:09:34.249704 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:34.249402 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-serving-cert: secret "maas-api-e2e-inf-a-780077-serving-cert" not found Oct 30 01:09:34.249704 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:34.249500 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls podName:a1a1e09b-20ef-4a61-833e-48e94dd4bbc6 nodeName:}" failed. No retries permitted until 2026-10-30 01:09:35.249471852 +0000 UTC m=+2113.821398054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls") pod "maas-api-e2e-inf-a-780077-7f49667468-6znm6" (UID: "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6") : secret "maas-api-e2e-inf-a-780077-serving-cert" not found Oct 30 01:09:34.453337 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:34.453298 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:09:34.502803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.502735 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:09:34.657153 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.657116 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-kube-api-access-gxmg8\" (UniqueName: \"kubernetes.io/projected/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-kube-api-access-gxmg8\") pod \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " Oct 30 01:09:34.657609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.657268 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-metrics-certs\") pod \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " Oct 30 01:09:34.657609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.657333 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls\") pod \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " Oct 30 01:09:34.657609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.657370 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-openshift-service-ca\") pod \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\" (UID: \"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6\") " Oct 30 01:09:34.657841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.657810 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-openshift-service-ca" pod "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" (UID: "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:34.658958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.658723 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:34.659981 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.659956 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-metrics-certs" pod "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" (UID: "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:34.660084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.659977 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls" pod "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" (UID: "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:34.660137 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.660081 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-kube-api-access-gxmg8" pod "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" (UID: "a1a1e09b-20ef-4a61-833e-48e94dd4bbc6"). InnerVolumeSpecName "kube-api-access-gxmg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:34.771294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.765935 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxmg8\" (UniqueName: \"kubernetes.io/projected/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-kube-api-access-gxmg8\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:34.771294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.765975 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:34.771294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.765988 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:34.973298 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.973259 4927 generic.go:356] "Generic (PLEG): container finished" podID="a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" containerID="3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293" exitCode=0 Oct 30 01:09:34.973298 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.973312 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" Oct 30 01:09:34.973784 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.973358 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" event={"ID":"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6","Type":"ContainerDied","Data":"3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293"} Oct 30 01:09:34.973784 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.973434 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6" event={"ID":"a1a1e09b-20ef-4a61-833e-48e94dd4bbc6","Type":"ContainerDied","Data":"22c479867d1a9fd38ea9fb81d6fbc2ce23aa6de9015678c5017c28f579b10033"} Oct 30 01:09:34.973784 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.973465 4927 scope.go:122] "RemoveContainer" containerID="3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293" Oct 30 01:09:34.986866 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.986666 4927 scope.go:122] "RemoveContainer" containerID="3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293" Oct 30 01:09:34.986963 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:34.986943 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293\": container with ID starting with 3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293 not found: ID does not exist" containerID="3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293" Oct 30 01:09:34.987006 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:34.986971 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293"} err="failed to get container status \"3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293\": rpc error: code = NotFound desc = could not find container \"3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293\": container with ID starting with 3f6fce37258efa603989ce135faa1dacf9928f2247e4fc3a2862bbe2d34b1293 not found: ID does not exist" Oct 30 01:09:35.025462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.025355 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6"] Oct 30 01:09:35.035872 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.035849 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-780077-7f49667468-6znm6"] Oct 30 01:09:35.560596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.557004 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:09:35.642674 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.631529 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" path="/var/lib/kubelet/pods/a1a1e09b-20ef-4a61-833e-48e94dd4bbc6/volumes" Oct 30 01:09:35.729822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.729787 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-kube-api-access-vwjpk\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-kube-api-access-vwjpk\") pod \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " Oct 30 01:09:35.730273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.729848 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-credential-socket\") pod \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " Oct 30 01:09:35.730273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.729958 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-data\") pod \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " Oct 30 01:09:35.730273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730041 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-certs\") pod \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " Oct 30 01:09:35.730273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730188 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-credential-socket" pod "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" (UID: "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.730273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730200 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-data" pod "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" (UID: "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.730565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730295 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-envoy\") pod \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " Oct 30 01:09:35.730565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730315 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-certs" pod "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" (UID: "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.730565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730344 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-token\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-token\") pod \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " Oct 30 01:09:35.730565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730463 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-socket\") pod \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " Oct 30 01:09:35.730565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730529 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istiod-ca-cert\") pod \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " Oct 30 01:09:35.730565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730588 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-podinfo\") pod \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\" (UID: \"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d\") " Oct 30 01:09:35.730851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730763 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-socket" pod "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" (UID: "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.730951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.730930 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istiod-ca-cert" pod "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" (UID: "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:35.731517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.731485 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.731517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.731516 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.731752 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.731530 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.731752 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.731545 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.731752 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.731564 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.732760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.732736 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-token" pod "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" (UID: "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:35.732888 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.732867 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-podinfo" pod "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" (UID: "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:35.733005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.732988 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-envoy" pod "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" (UID: "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:35.733076 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.733061 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-kube-api-access-vwjpk" pod "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" (UID: "1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d"). InnerVolumeSpecName "kube-api-access-vwjpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:35.833677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.833641 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.833677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.833669 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.833677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.833679 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.833906 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.833687 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwjpk\" (UniqueName: \"kubernetes.io/projected/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d-kube-api-access-vwjpk\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:35.983651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.983534 4927 generic.go:356] "Generic (PLEG): container finished" podID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerID="9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84" exitCode=0 Oct 30 01:09:35.983651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.983609 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" Oct 30 01:09:35.983651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.983619 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" event={"ID":"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d","Type":"ContainerDied","Data":"9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84"} Oct 30 01:09:35.983881 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.983680 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr" event={"ID":"1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d","Type":"ContainerDied","Data":"2707fe73a1d0da4d5813a47b5c6d723b883fdaecbdc6ed1f35ae666bb32590d6"} Oct 30 01:09:35.983881 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.983709 4927 scope.go:122] "RemoveContainer" containerID="9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84" Oct 30 01:09:35.997055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:35.996880 4927 scope.go:122] "RemoveContainer" containerID="f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6" Oct 30 01:09:36.007306 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:36.007287 4927 scope.go:122] "RemoveContainer" containerID="9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84" Oct 30 01:09:36.007597 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:36.007549 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84\": container with ID starting with 9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84 not found: ID does not exist" containerID="9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84" Oct 30 01:09:36.007672 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:36.007613 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84"} err="failed to get container status \"9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84\": rpc error: code = NotFound desc = could not find container \"9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84\": container with ID starting with 9e438f27a0a940c1d0b3570c3830d05301a94966edfea2cf0be6320cf88deb84 not found: ID does not exist" Oct 30 01:09:36.007672 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:36.007640 4927 scope.go:122] "RemoveContainer" containerID="f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6" Oct 30 01:09:36.007913 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:36.007893 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6\": container with ID starting with f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6 not found: ID does not exist" containerID="f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6" Oct 30 01:09:36.007953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:36.007920 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6"} err="failed to get container status \"f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6\": rpc error: code = NotFound desc = could not find container \"f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6\": container with ID starting with f83285360a8a7383cb33965c597c1a778346cac3b61d90fb03278780965a2eb6 not found: ID does not exist" Oct 30 01:09:36.048663 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:36.048638 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr"] Oct 30 01:09:36.060743 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:36.060707 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dd257f-openshift-default-748b4d89df-n7nvr"] Oct 30 01:09:37.674595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:37.674382 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" path="/var/lib/kubelet/pods/1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d/volumes" Oct 30 01:09:38.388721 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:38.384590 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s"] Oct 30 01:09:40.066614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.066561 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" podUID="83d079dd-5919-43c0-b623-f17e7b1bd38d" containerName="main" containerID="cri-o://c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e" gracePeriod=45 Oct 30 01:09:40.422453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.422421 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:09:40.454831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.454803 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-model-cache\") pod \"83d079dd-5919-43c0-b623-f17e7b1bd38d\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " Oct 30 01:09:40.454992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.454843 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-dshm\") pod \"83d079dd-5919-43c0-b623-f17e7b1bd38d\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " Oct 30 01:09:40.454992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.454882 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83d079dd-5919-43c0-b623-f17e7b1bd38d-tls-certs\" (UniqueName: \"kubernetes.io/secret/83d079dd-5919-43c0-b623-f17e7b1bd38d-tls-certs\") pod \"83d079dd-5919-43c0-b623-f17e7b1bd38d\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " Oct 30 01:09:40.454992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.454923 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-home\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-home\") pod \"83d079dd-5919-43c0-b623-f17e7b1bd38d\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " Oct 30 01:09:40.455157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.455023 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d079dd-5919-43c0-b623-f17e7b1bd38d-kube-api-access-8ctc6\" (UniqueName: \"kubernetes.io/projected/83d079dd-5919-43c0-b623-f17e7b1bd38d-kube-api-access-8ctc6\") pod \"83d079dd-5919-43c0-b623-f17e7b1bd38d\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " Oct 30 01:09:40.455157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.455071 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-tmp-dir\") pod \"83d079dd-5919-43c0-b623-f17e7b1bd38d\" (UID: \"83d079dd-5919-43c0-b623-f17e7b1bd38d\") " Oct 30 01:09:40.455157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.455111 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-model-cache" pod "83d079dd-5919-43c0-b623-f17e7b1bd38d" (UID: "83d079dd-5919-43c0-b623-f17e7b1bd38d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.455313 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.455275 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-home" pod "83d079dd-5919-43c0-b623-f17e7b1bd38d" (UID: "83d079dd-5919-43c0-b623-f17e7b1bd38d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.455431 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.455408 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-tmp-dir" pod "83d079dd-5919-43c0-b623-f17e7b1bd38d" (UID: "83d079dd-5919-43c0-b623-f17e7b1bd38d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.455857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.455838 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.455857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.455858 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.455987 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.455866 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.457222 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.457203 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d079dd-5919-43c0-b623-f17e7b1bd38d-tls-certs" pod "83d079dd-5919-43c0-b623-f17e7b1bd38d" (UID: "83d079dd-5919-43c0-b623-f17e7b1bd38d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:40.457527 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.457500 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-dshm" pod "83d079dd-5919-43c0-b623-f17e7b1bd38d" (UID: "83d079dd-5919-43c0-b623-f17e7b1bd38d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.457527 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.457518 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d079dd-5919-43c0-b623-f17e7b1bd38d-kube-api-access-8ctc6" pod "83d079dd-5919-43c0-b623-f17e7b1bd38d" (UID: "83d079dd-5919-43c0-b623-f17e7b1bd38d"). InnerVolumeSpecName "kube-api-access-8ctc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:40.505785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.505740 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" podUID="b99f3492-5f82-45bf-a645-9f28cad2a971" containerName="main" containerID="cri-o://f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692" gracePeriod=45 Oct 30 01:09:40.556920 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.556891 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d079dd-5919-43c0-b623-f17e7b1bd38d-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.556920 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.556919 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d079dd-5919-43c0-b623-f17e7b1bd38d-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.557091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.556930 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ctc6\" (UniqueName: \"kubernetes.io/projected/83d079dd-5919-43c0-b623-f17e7b1bd38d-kube-api-access-8ctc6\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.820620 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.820563 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:09:40.861703 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.861676 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-tmp-dir\") pod \"b99f3492-5f82-45bf-a645-9f28cad2a971\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " Oct 30 01:09:40.861871 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.861733 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99f3492-5f82-45bf-a645-9f28cad2a971-tls-certs\" (UniqueName: \"kubernetes.io/secret/b99f3492-5f82-45bf-a645-9f28cad2a971-tls-certs\") pod \"b99f3492-5f82-45bf-a645-9f28cad2a971\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " Oct 30 01:09:40.861871 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.861814 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-home\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-home\") pod \"b99f3492-5f82-45bf-a645-9f28cad2a971\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " Oct 30 01:09:40.862011 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.861971 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-tmp-dir" pod "b99f3492-5f82-45bf-a645-9f28cad2a971" (UID: "b99f3492-5f82-45bf-a645-9f28cad2a971"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.862078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.862031 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-model-cache\") pod \"b99f3492-5f82-45bf-a645-9f28cad2a971\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " Oct 30 01:09:40.862131 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.862082 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-home" pod "b99f3492-5f82-45bf-a645-9f28cad2a971" (UID: "b99f3492-5f82-45bf-a645-9f28cad2a971"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.862174 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.862161 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-dshm\") pod \"b99f3492-5f82-45bf-a645-9f28cad2a971\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " Oct 30 01:09:40.862247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.862232 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99f3492-5f82-45bf-a645-9f28cad2a971-kube-api-access-pbshg\" (UniqueName: \"kubernetes.io/projected/b99f3492-5f82-45bf-a645-9f28cad2a971-kube-api-access-pbshg\") pod \"b99f3492-5f82-45bf-a645-9f28cad2a971\" (UID: \"b99f3492-5f82-45bf-a645-9f28cad2a971\") " Oct 30 01:09:40.862293 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.862271 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-model-cache" pod "b99f3492-5f82-45bf-a645-9f28cad2a971" (UID: "b99f3492-5f82-45bf-a645-9f28cad2a971"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.863028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.862993 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.863028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.863020 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.863236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.863036 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.864491 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.864463 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-dshm" pod "b99f3492-5f82-45bf-a645-9f28cad2a971" (UID: "b99f3492-5f82-45bf-a645-9f28cad2a971"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.864660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.864561 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99f3492-5f82-45bf-a645-9f28cad2a971-tls-certs" pod "b99f3492-5f82-45bf-a645-9f28cad2a971" (UID: "b99f3492-5f82-45bf-a645-9f28cad2a971"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:40.864660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.864641 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99f3492-5f82-45bf-a645-9f28cad2a971-kube-api-access-pbshg" pod "b99f3492-5f82-45bf-a645-9f28cad2a971" (UID: "b99f3492-5f82-45bf-a645-9f28cad2a971"). InnerVolumeSpecName "kube-api-access-pbshg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:40.964217 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.964135 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbshg\" (UniqueName: \"kubernetes.io/projected/b99f3492-5f82-45bf-a645-9f28cad2a971-kube-api-access-pbshg\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.964217 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.964164 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99f3492-5f82-45bf-a645-9f28cad2a971-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.964217 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:40.964174 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99f3492-5f82-45bf-a645-9f28cad2a971-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:41.215380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.215281 4927 generic.go:356] "Generic (PLEG): container finished" podID="83d079dd-5919-43c0-b623-f17e7b1bd38d" containerID="c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e" exitCode=0 Oct 30 01:09:41.215380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.215341 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" Oct 30 01:09:41.215380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.215357 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" event={"ID":"83d079dd-5919-43c0-b623-f17e7b1bd38d","Type":"ContainerDied","Data":"c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e"} Oct 30 01:09:41.216000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.215396 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw" event={"ID":"83d079dd-5919-43c0-b623-f17e7b1bd38d","Type":"ContainerDied","Data":"1ef2cf9a45a2a66ee10d652d42c1cef6016bc2c3b7e7e46e0f35f9b656379e43"} Oct 30 01:09:41.216000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.215417 4927 scope.go:122] "RemoveContainer" containerID="c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e" Oct 30 01:09:41.217382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.217352 4927 generic.go:356] "Generic (PLEG): container finished" podID="b99f3492-5f82-45bf-a645-9f28cad2a971" containerID="f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692" exitCode=0 Oct 30 01:09:41.217508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.217476 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" event={"ID":"b99f3492-5f82-45bf-a645-9f28cad2a971","Type":"ContainerDied","Data":"f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692"} Oct 30 01:09:41.217565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.217509 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" event={"ID":"b99f3492-5f82-45bf-a645-9f28cad2a971","Type":"ContainerDied","Data":"bd4fa8c3136052ebb94a07a53e42a7b02adf6503c1b6e58c60f1d561666e7683"} Oct 30 01:09:41.217680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.217625 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm" Oct 30 01:09:41.229354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.229335 4927 scope.go:122] "RemoveContainer" containerID="c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e" Oct 30 01:09:41.229653 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:41.229632 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e\": container with ID starting with c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e not found: ID does not exist" containerID="c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e" Oct 30 01:09:41.229758 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.229658 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e"} err="failed to get container status \"c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e\": rpc error: code = NotFound desc = could not find container \"c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e\": container with ID starting with c604039812d3aa69dfd46235d3bb67266a69f96a0c16250c41dfae9122c6c60e not found: ID does not exist" Oct 30 01:09:41.229758 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.229676 4927 scope.go:122] "RemoveContainer" containerID="f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692" Oct 30 01:09:41.239886 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.239867 4927 scope.go:122] "RemoveContainer" containerID="f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692" Oct 30 01:09:41.240128 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:41.240109 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692\": container with ID starting with f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692 not found: ID does not exist" containerID="f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692" Oct 30 01:09:41.240189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.240138 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692"} err="failed to get container status \"f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692\": rpc error: code = NotFound desc = could not find container \"f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692\": container with ID starting with f1baac2470fc5f13bca83e13e91f96fa4c6e24ae1b13e8bb923fbb0ec5ce5692 not found: ID does not exist" Oct 30 01:09:41.269286 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.269264 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw"] Oct 30 01:09:41.299596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.298982 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/auth-test-model-969b92-kserve-7b55c9748f-pbhxw"] Oct 30 01:09:41.326312 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.326272 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm"] Oct 30 01:09:41.331879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.331852 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cf3466/test-model-cf3466-kserve-5c68988b55-6stcm"] Oct 30 01:09:41.977649 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.974501 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d079dd-5919-43c0-b623-f17e7b1bd38d" path="/var/lib/kubelet/pods/83d079dd-5919-43c0-b623-f17e7b1bd38d/volumes" Oct 30 01:09:41.979426 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:41.979379 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99f3492-5f82-45bf-a645-9f28cad2a971" path="/var/lib/kubelet/pods/b99f3492-5f82-45bf-a645-9f28cad2a971/volumes" Oct 30 01:09:43.393722 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:43.393668 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" podUID="865e10fe-0a6f-40bf-84c4-5f8b0153d891" containerName="payload-pre-processing" containerID="cri-o://be7a8243d58c8efd71db81a54684720e34621a68415235675125bb8f650b2d45" gracePeriod=25 Oct 30 01:09:44.400330 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.400281 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" podUID="7ea7d8b4-a022-4240-a420-b2443bca2a03" containerName="main" containerID="cri-o://c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38" gracePeriod=45 Oct 30 01:09:44.695817 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.695786 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:09:44.744312 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.744275 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea7d8b4-a022-4240-a420-b2443bca2a03-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea7d8b4-a022-4240-a420-b2443bca2a03-tls-certs\") pod \"7ea7d8b4-a022-4240-a420-b2443bca2a03\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " Oct 30 01:09:44.744601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.744337 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-tmp-dir\") pod \"7ea7d8b4-a022-4240-a420-b2443bca2a03\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " Oct 30 01:09:44.744601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.744443 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-model-cache\") pod \"7ea7d8b4-a022-4240-a420-b2443bca2a03\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " Oct 30 01:09:44.744601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.744506 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-dshm\") pod \"7ea7d8b4-a022-4240-a420-b2443bca2a03\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " Oct 30 01:09:44.744601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.744566 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea7d8b4-a022-4240-a420-b2443bca2a03-kube-api-access-8stp2\" (UniqueName: \"kubernetes.io/projected/7ea7d8b4-a022-4240-a420-b2443bca2a03-kube-api-access-8stp2\") pod \"7ea7d8b4-a022-4240-a420-b2443bca2a03\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " Oct 30 01:09:44.749319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.744753 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-home\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-home\") pod \"7ea7d8b4-a022-4240-a420-b2443bca2a03\" (UID: \"7ea7d8b4-a022-4240-a420-b2443bca2a03\") " Oct 30 01:09:44.749319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.745682 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-home" pod "7ea7d8b4-a022-4240-a420-b2443bca2a03" (UID: "7ea7d8b4-a022-4240-a420-b2443bca2a03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:44.749319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.746365 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-tmp-dir" pod "7ea7d8b4-a022-4240-a420-b2443bca2a03" (UID: "7ea7d8b4-a022-4240-a420-b2443bca2a03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:44.749319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.746888 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-model-cache" pod "7ea7d8b4-a022-4240-a420-b2443bca2a03" (UID: "7ea7d8b4-a022-4240-a420-b2443bca2a03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:44.751241 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.751201 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea7d8b4-a022-4240-a420-b2443bca2a03-kube-api-access-8stp2" pod "7ea7d8b4-a022-4240-a420-b2443bca2a03" (UID: "7ea7d8b4-a022-4240-a420-b2443bca2a03"). InnerVolumeSpecName "kube-api-access-8stp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:44.754005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.752846 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-dshm" pod "7ea7d8b4-a022-4240-a420-b2443bca2a03" (UID: "7ea7d8b4-a022-4240-a420-b2443bca2a03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:44.762424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.762396 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea7d8b4-a022-4240-a420-b2443bca2a03-tls-certs" pod "7ea7d8b4-a022-4240-a420-b2443bca2a03" (UID: "7ea7d8b4-a022-4240-a420-b2443bca2a03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:44.847553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.847470 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.847553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.847505 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea7d8b4-a022-4240-a420-b2443bca2a03-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.847553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.847522 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.847553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.847535 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.847553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.847549 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea7d8b4-a022-4240-a420-b2443bca2a03-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.847844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:44.847563 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8stp2\" (UniqueName: \"kubernetes.io/projected/7ea7d8b4-a022-4240-a420-b2443bca2a03-kube-api-access-8stp2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:09:45.313440 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.313396 4927 generic.go:356] "Generic (PLEG): container finished" podID="7ea7d8b4-a022-4240-a420-b2443bca2a03" containerID="c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38" exitCode=0 Oct 30 01:09:45.313742 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.313720 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" event={"ID":"7ea7d8b4-a022-4240-a420-b2443bca2a03","Type":"ContainerDied","Data":"c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38"} Oct 30 01:09:45.313861 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.313767 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" event={"ID":"7ea7d8b4-a022-4240-a420-b2443bca2a03","Type":"ContainerDied","Data":"db25cec603d45714e5197745a36c480471e07cd8bc27260c3c69ae14bcbc3b97"} Oct 30 01:09:45.313861 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.313792 4927 scope.go:122] "RemoveContainer" containerID="c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38" Oct 30 01:09:45.313993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.313977 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz" Oct 30 01:09:45.339609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.339554 4927 scope.go:122] "RemoveContainer" containerID="c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38" Oct 30 01:09:45.339958 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:09:45.339933 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38\": container with ID starting with c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38 not found: ID does not exist" containerID="c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38" Oct 30 01:09:45.340055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.339968 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38"} err="failed to get container status \"c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38\": rpc error: code = NotFound desc = could not find container \"c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38\": container with ID starting with c9ef443430f97396db6f47d382c9831c4a0ff460b7b5d557ec8fd8d6c8f47f38 not found: ID does not exist" Oct 30 01:09:45.376452 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.376255 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz"] Oct 30 01:09:45.380851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.380821 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/auth-test-model-ae06fe-kserve-66996467d9-qsgxz"] Oct 30 01:09:45.573777 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:45.573702 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea7d8b4-a022-4240-a420-b2443bca2a03" path="/var/lib/kubelet/pods/7ea7d8b4-a022-4240-a420-b2443bca2a03/volumes" Oct 30 01:09:46.791550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:46.791498 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" podUID="865e10fe-0a6f-40bf-84c4-5f8b0153d891" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:09:49.375844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:49.375802 4927 generic.go:356] "Generic (PLEG): container finished" podID="ee87ca52-c410-4f3b-9906-ff823188c656" containerID="836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d" exitCode=0 Oct 30 01:09:49.376317 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:49.375882 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" event={"ID":"ee87ca52-c410-4f3b-9906-ff823188c656","Type":"ContainerDied","Data":"836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d"} Oct 30 01:09:49.376469 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:49.376435 4927 scope.go:122] "RemoveContainer" containerID="836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d" Oct 30 01:09:49.378272 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:49.378239 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:49.378347 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:49.378314 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:49.378387 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:49.378346 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:50.414142 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:50.414104 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" event={"ID":"ee87ca52-c410-4f3b-9906-ff823188c656","Type":"ContainerStarted","Data":"3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1"} Oct 30 01:09:50.659945 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:50.659914 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 01:09:50.661633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:50.661604 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 01:09:50.661747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:50.661659 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 01:09:51.891699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:51.891660 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 01:09:51.892416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:51.892376 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 01:09:52.660878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:52.660839 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 01:09:52.661069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:52.660909 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 01:09:53.406950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:53.406911 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd"] Oct 30 01:09:53.661630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:53.661112 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 01:09:53.661630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:53.661189 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 01:09:54.659837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:54.659798 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 01:09:54.660320 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:54.659860 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 01:09:55.296746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:55.296709 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 01:09:55.660425 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:55.660384 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 01:09:55.660890 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:55.660438 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 01:09:56.718771 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:56.718736 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 01:09:56.723517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:56.723488 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 01:09:56.789480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:56.789446 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" podUID="865e10fe-0a6f-40bf-84c4-5f8b0153d891" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:09:58.136137 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136101 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms"] Oct 30 01:09:58.136928 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136912 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" Oct 30 01:09:58.136983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136929 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" Oct 30 01:09:58.136983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136939 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" Oct 30 01:09:58.136983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136944 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" Oct 30 01:09:58.136983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136959 4927 cpu_manager.go:414] "containerMap: removing container" podUID="365d784b-a986-49a5-ba10-93a80bdf2630" containerName="main" Oct 30 01:09:58.136983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136964 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365d784b-a986-49a5-ba10-93a80bdf2630" containerName="main" Oct 30 01:09:58.136983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136973 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6a54e8b1-b885-49c6-b803-5cad96e73314" containerName="main" Oct 30 01:09:58.136983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136978 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a54e8b1-b885-49c6-b803-5cad96e73314" containerName="main" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136994 4927 cpu_manager.go:414] "containerMap: removing container" podUID="83d079dd-5919-43c0-b623-f17e7b1bd38d" containerName="main" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.136999 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d079dd-5919-43c0-b623-f17e7b1bd38d" containerName="main" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137011 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3ffc62a3-2f1e-475d-9073-2ecab906fe07" containerName="payload-processing" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137016 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ffc62a3-2f1e-475d-9073-2ecab906fe07" containerName="payload-processing" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137027 4927 cpu_manager.go:414] "containerMap: removing container" podUID="faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" containerName="main" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137031 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" containerName="main" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137043 4927 cpu_manager.go:414] "containerMap: removing container" podUID="92fe5a50-15bf-4ff7-ab5b-9540ce501400" containerName="maas-api" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137047 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92fe5a50-15bf-4ff7-ab5b-9540ce501400" containerName="maas-api" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137054 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137059 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137067 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137072 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137083 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7ea7d8b4-a022-4240-a420-b2443bca2a03" containerName="main" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137087 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea7d8b4-a022-4240-a420-b2443bca2a03" containerName="main" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137098 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137102 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137107 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1e0c2d70-f1da-49bb-8969-3c78aad572b1" containerName="maas-api" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137112 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0c2d70-f1da-49bb-8969-3c78aad572b1" containerName="maas-api" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137122 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" containerName="maas-api" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137126 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" containerName="maas-api" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137132 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137137 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137154 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137159 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137165 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b99f3492-5f82-45bf-a645-9f28cad2a971" containerName="main" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137170 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99f3492-5f82-45bf-a645-9f28cad2a971" containerName="main" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137176 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" Oct 30 01:09:58.137204 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137180 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137285 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a54e8b1-b885-49c6-b803-5cad96e73314" containerName="main" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137294 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d079dd-5919-43c0-b623-f17e7b1bd38d" containerName="main" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137301 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ffc62a3-2f1e-475d-9073-2ecab906fe07" containerName="payload-processing" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137307 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa103dd-7f9a-43b8-bbbe-97ca9e80bdc8" containerName="main" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137315 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="92fe5a50-15bf-4ff7-ab5b-9540ce501400" containerName="maas-api" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137323 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137329 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5045c19-6086-4a68-801c-e3f7b3c098a3" containerName="istio-proxy" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137335 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea7d8b4-a022-4240-a420-b2443bca2a03" containerName="main" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137343 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef083d76-7f00-49e0-b7d4-657f48389486" containerName="istio-proxy" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137348 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0c2d70-f1da-49bb-8969-3c78aad572b1" containerName="maas-api" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137355 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a1e09b-20ef-4a61-833e-48e94dd4bbc6" containerName="maas-api" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137359 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5b9fe3-ac8b-4727-99ac-47a588050de8" containerName="istio-proxy" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137368 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137374 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99f3492-5f82-45bf-a645-9f28cad2a971" containerName="main" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137379 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2e24bf-32fb-44c8-a50c-d0b2abacdc5d" containerName="istio-proxy" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137386 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137390 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2292a1de-9dcc-4d12-bf6c-8334bc4a8b6a" containerName="istio-proxy" Oct 30 01:09:58.138049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.137394 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="365d784b-a986-49a5-ba10-93a80bdf2630" containerName="main" Oct 30 01:09:58.141564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.141548 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.163130 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.163106 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms"] Oct 30 01:09:58.168741 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.168716 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-token\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.168944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.168754 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istiod-ca-cert\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.168944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.168878 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-certs\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.168944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.168908 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-credential-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.169086 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.169065 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-podinfo\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.169233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.169205 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.169361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.169254 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-data\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.169361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.169268 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-97ceef-openshift-default-dockercfg-zqr8k\"" Oct 30 01:09:58.169361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.169283 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-envoy\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.169542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.169373 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9kb\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-kube-api-access-mm9kb\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.270482 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.270444 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-podinfo\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.270699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.270537 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.270699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.270566 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-data\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.270699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.270625 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-envoy\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.270699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.270666 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9kb\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-kube-api-access-mm9kb\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.270959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.270840 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-token\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.270959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.270866 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istiod-ca-cert\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.270959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.270930 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-certs\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.271114 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.270958 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-credential-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.271114 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.271039 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.271114 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.271056 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-data\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.271347 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.271327 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-credential-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.271416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.271387 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-certs\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.271700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.271678 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istiod-ca-cert\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.273105 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.273077 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-envoy\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.273326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.273310 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-podinfo\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.293078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.293054 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-token\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.293194 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.293175 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9kb\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-kube-api-access-mm9kb\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.407740 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.407625 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" podUID="facaba60-f652-49ae-8371-8f69798f1c53" containerName="payload-processing" containerID="cri-o://a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57" gracePeriod=25 Oct 30 01:09:58.456700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.456659 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:09:58.820373 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:09:58.820341 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4656a36_4d2a_4e66_969d_7f3d7b5c9358.slice/crio-49b0b8a3d13ca053603492c57fbce96161a456a81fa2b6c931965657b41e9dc7 WatchSource:0}: Error finding container 49b0b8a3d13ca053603492c57fbce96161a456a81fa2b6c931965657b41e9dc7: Status 404 returned error can't find the container with id 49b0b8a3d13ca053603492c57fbce96161a456a81fa2b6c931965657b41e9dc7 Oct 30 01:09:58.822777 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.822744 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms"] Oct 30 01:09:58.824156 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.824113 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:58.824267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.824209 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:58.824348 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:58.824263 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:59.510650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:59.510608 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" event={"ID":"e4656a36-4d2a-4e66-969d-7f3d7b5c9358","Type":"ContainerStarted","Data":"416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351"} Oct 30 01:09:59.511093 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:59.510670 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" event={"ID":"e4656a36-4d2a-4e66-969d-7f3d7b5c9358","Type":"ContainerStarted","Data":"49b0b8a3d13ca053603492c57fbce96161a456a81fa2b6c931965657b41e9dc7"} Oct 30 01:09:59.605463 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:09:59.605405 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" podStartSLOduration=1.605389935 podStartE2EDuration="1.605389935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:59.566876972 +0000 UTC m=+2138.138803195" watchObservedRunningTime="2026-10-30 01:09:59.605389935 +0000 UTC m=+2138.177316586" Oct 30 01:10:00.457051 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:00.457016 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:10:00.458407 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:00.458383 4927 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:10:00.458518 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:00.458428 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 01:10:01.457091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:01.457051 4927 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:10:01.457525 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:01.457108 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 01:10:02.119120 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.119084 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8"] Oct 30 01:10:02.124372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.124347 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.147461 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.147429 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8"] Oct 30 01:10:02.149292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.149254 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-bea3ef\"/\"kube-root-ca.crt\"" Oct 30 01:10:02.149292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.149265 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-bea3ef\"/\"auth-test-model-bea3ef-kserve-self-signed-certs\"" Oct 30 01:10:02.149512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.149267 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-bea3ef\"/\"openshift-service-ca.crt\"" Oct 30 01:10:02.150519 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.150498 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-bea3ef\"/\"default-dockercfg-hq45k\"" Oct 30 01:10:02.155544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.155516 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-model-cache\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.155670 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.155565 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tmp-dir\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.155793 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.155772 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-dshm\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.155904 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.155885 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-home\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.156056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.156040 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwccc\" (UniqueName: \"kubernetes.io/projected/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-kube-api-access-gwccc\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.156133 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.156074 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tls-certs\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.257923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.257885 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-dshm\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.258074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.257981 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-home\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.258136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.258086 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwccc\" (UniqueName: \"kubernetes.io/projected/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-kube-api-access-gwccc\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.258136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.258115 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tls-certs\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.258240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.258169 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-model-cache\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.258240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.258191 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tmp-dir\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.258391 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.258362 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-home\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.258556 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.258532 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-model-cache\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.258654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.258620 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tmp-dir\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.260240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.260215 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-dshm\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.260606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.260588 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tls-certs\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.303595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.300650 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwccc\" (UniqueName: \"kubernetes.io/projected/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-kube-api-access-gwccc\") pod \"auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.497592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.496958 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:02.521953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.521913 4927 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:10:02.522255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:02.522211 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 01:10:03.040937 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.040908 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8"] Oct 30 01:10:03.041598 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:10:03.041556 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae44dd37_5d61_4662_a6cb_6cbc7b369d58.slice/crio-f920b05826d91ab3db54c871ad9a0ffba42de3e6b4a10ff04c35420fea4bb31a WatchSource:0}: Error finding container f920b05826d91ab3db54c871ad9a0ffba42de3e6b4a10ff04c35420fea4bb31a: Status 404 returned error can't find the container with id f920b05826d91ab3db54c871ad9a0ffba42de3e6b4a10ff04c35420fea4bb31a Oct 30 01:10:03.457916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.457883 4927 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:10:03.458062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.457948 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 01:10:03.649330 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.649293 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" event={"ID":"ae44dd37-5d61-4662-a6cb-6cbc7b369d58","Type":"ContainerStarted","Data":"f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c"} Oct 30 01:10:03.649686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.649343 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" event={"ID":"ae44dd37-5d61-4662-a6cb-6cbc7b369d58","Type":"ContainerStarted","Data":"f920b05826d91ab3db54c871ad9a0ffba42de3e6b4a10ff04c35420fea4bb31a"} Oct 30 01:10:03.651026 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.650994 4927 generic.go:356] "Generic (PLEG): container finished" podID="865e10fe-0a6f-40bf-84c4-5f8b0153d891" containerID="be7a8243d58c8efd71db81a54684720e34621a68415235675125bb8f650b2d45" exitCode=0 Oct 30 01:10:03.651136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.651057 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" event={"ID":"865e10fe-0a6f-40bf-84c4-5f8b0153d891","Type":"ContainerDied","Data":"be7a8243d58c8efd71db81a54684720e34621a68415235675125bb8f650b2d45"} Oct 30 01:10:03.698845 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.692475 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" podStartSLOduration=1.69245501 podStartE2EDuration="1.69245501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:03.690101994 +0000 UTC m=+2142.262028229" watchObservedRunningTime="2026-10-30 01:10:03.69245501 +0000 UTC m=+2142.264381233" Oct 30 01:10:03.709677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.709650 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:10:03.723406 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.723378 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865e10fe-0a6f-40bf-84c4-5f8b0153d891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865e10fe-0a6f-40bf-84c4-5f8b0153d891-plugins-config-volume\") pod \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\" (UID: \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\") " Oct 30 01:10:03.723683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.723663 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865e10fe-0a6f-40bf-84c4-5f8b0153d891-kube-api-access-rxb5q\" (UniqueName: \"kubernetes.io/projected/865e10fe-0a6f-40bf-84c4-5f8b0153d891-kube-api-access-rxb5q\") pod \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\" (UID: \"865e10fe-0a6f-40bf-84c4-5f8b0153d891\") " Oct 30 01:10:03.723777 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.723755 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865e10fe-0a6f-40bf-84c4-5f8b0153d891-plugins-config-volume" pod "865e10fe-0a6f-40bf-84c4-5f8b0153d891" (UID: "865e10fe-0a6f-40bf-84c4-5f8b0153d891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:03.724280 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.724261 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865e10fe-0a6f-40bf-84c4-5f8b0153d891-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:03.725790 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.725769 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865e10fe-0a6f-40bf-84c4-5f8b0153d891-kube-api-access-rxb5q" pod "865e10fe-0a6f-40bf-84c4-5f8b0153d891" (UID: "865e10fe-0a6f-40bf-84c4-5f8b0153d891"). InnerVolumeSpecName "kube-api-access-rxb5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:03.839607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:03.834030 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxb5q\" (UniqueName: \"kubernetes.io/projected/865e10fe-0a6f-40bf-84c4-5f8b0153d891-kube-api-access-rxb5q\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:04.105033 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.104941 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch"] Oct 30 01:10:04.105224 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.105183 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" podUID="6a6cc10e-46b7-4122-a64f-ed718babf1b8" containerName="maas-api" containerID="cri-o://635f134a15a3bd512c130464a9967b28e46d002889595f05674e49e4104a0b61" gracePeriod=30 Oct 30 01:10:04.467607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.462092 4927 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:10:04.467607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.462167 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 01:10:04.500257 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.500216 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:04.858814 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.858785 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" event={"ID":"865e10fe-0a6f-40bf-84c4-5f8b0153d891","Type":"ContainerDied","Data":"cddbe6833bc58185e7b4fd0f7be29015db5c2a5e1a34d2ee5383196c3b2921f8"} Oct 30 01:10:04.859222 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.859208 4927 scope.go:122] "RemoveContainer" containerID="be7a8243d58c8efd71db81a54684720e34621a68415235675125bb8f650b2d45" Oct 30 01:10:04.859748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.859730 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s" Oct 30 01:10:04.869107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.869058 4927 generic.go:356] "Generic (PLEG): container finished" podID="6a6cc10e-46b7-4122-a64f-ed718babf1b8" containerID="635f134a15a3bd512c130464a9967b28e46d002889595f05674e49e4104a0b61" exitCode=0 Oct 30 01:10:04.869425 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.869400 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" event={"ID":"6a6cc10e-46b7-4122-a64f-ed718babf1b8","Type":"ContainerDied","Data":"635f134a15a3bd512c130464a9967b28e46d002889595f05674e49e4104a0b61"} Oct 30 01:10:04.904597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:04.894547 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:10:04.922245 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:10:04.922202 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:10:05.026654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.026614 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s"] Oct 30 01:10:05.038094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.038064 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-97ceef-86cf87f57f-9m55s"] Oct 30 01:10:05.046538 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.046510 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6cc10e-46b7-4122-a64f-ed718babf1b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6cc10e-46b7-4122-a64f-ed718babf1b8-openshift-service-ca\") pod \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " Oct 30 01:10:05.046714 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.046558 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6cc10e-46b7-4122-a64f-ed718babf1b8-kube-api-access-jpk5r\" (UniqueName: \"kubernetes.io/projected/6a6cc10e-46b7-4122-a64f-ed718babf1b8-kube-api-access-jpk5r\") pod \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " Oct 30 01:10:05.046714 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.046623 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-metrics-certs\") pod \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " Oct 30 01:10:05.046714 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.046675 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-maas-api-tls\") pod \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\" (UID: \"6a6cc10e-46b7-4122-a64f-ed718babf1b8\") " Oct 30 01:10:05.046985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.046960 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6cc10e-46b7-4122-a64f-ed718babf1b8-openshift-service-ca" pod "6a6cc10e-46b7-4122-a64f-ed718babf1b8" (UID: "6a6cc10e-46b7-4122-a64f-ed718babf1b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:05.047482 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.047453 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6cc10e-46b7-4122-a64f-ed718babf1b8-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:05.049069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.049041 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-maas-api-tls" pod "6a6cc10e-46b7-4122-a64f-ed718babf1b8" (UID: "6a6cc10e-46b7-4122-a64f-ed718babf1b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:05.049155 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.049067 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-metrics-certs" pod "6a6cc10e-46b7-4122-a64f-ed718babf1b8" (UID: "6a6cc10e-46b7-4122-a64f-ed718babf1b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:05.049155 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.049067 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6cc10e-46b7-4122-a64f-ed718babf1b8-kube-api-access-jpk5r" pod "6a6cc10e-46b7-4122-a64f-ed718babf1b8" (UID: "6a6cc10e-46b7-4122-a64f-ed718babf1b8"). InnerVolumeSpecName "kube-api-access-jpk5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:05.148833 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.148794 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpk5r\" (UniqueName: \"kubernetes.io/projected/6a6cc10e-46b7-4122-a64f-ed718babf1b8-kube-api-access-jpk5r\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:05.149010 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.148843 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:05.149010 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.148861 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6cc10e-46b7-4122-a64f-ed718babf1b8-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:05.573881 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.573801 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865e10fe-0a6f-40bf-84c4-5f8b0153d891" path="/var/lib/kubelet/pods/865e10fe-0a6f-40bf-84c4-5f8b0153d891/volumes" Oct 30 01:10:05.715486 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.715272 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:10:05.715857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.715821 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:10:05.718520 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.718501 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:10:05.884919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.884882 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" Oct 30 01:10:05.884919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.884904 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch" event={"ID":"6a6cc10e-46b7-4122-a64f-ed718babf1b8","Type":"ContainerDied","Data":"e3e4eb34789ebc2e790d58ddb56d9391251f06d500da3953de1ddc9219188111"} Oct 30 01:10:05.885489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.884984 4927 scope.go:122] "RemoveContainer" containerID="635f134a15a3bd512c130464a9967b28e46d002889595f05674e49e4104a0b61" Oct 30 01:10:05.948483 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.948437 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch"] Oct 30 01:10:05.952461 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:05.952438 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97ceef-9bf7794fc-2sjch"] Oct 30 01:10:06.201549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:06.201464 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" podUID="facaba60-f652-49ae-8371-8f69798f1c53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:07.586708 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:07.586667 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6cc10e-46b7-4122-a64f-ed718babf1b8" path="/var/lib/kubelet/pods/6a6cc10e-46b7-4122-a64f-ed718babf1b8/volumes" Oct 30 01:10:12.301807 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:12.301773 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms"] Oct 30 01:10:12.302343 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:12.301985 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerName="istio-proxy" containerID="cri-o://416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351" gracePeriod=30 Oct 30 01:10:12.543592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:12.534744 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:12.612728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:12.610546 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:13.062428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.062397 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:10:13.494941 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.494909 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx"] Oct 30 01:10:13.495776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.495748 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6a6cc10e-46b7-4122-a64f-ed718babf1b8" containerName="maas-api" Oct 30 01:10:13.495776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.495767 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6cc10e-46b7-4122-a64f-ed718babf1b8" containerName="maas-api" Oct 30 01:10:13.495973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.495791 4927 cpu_manager.go:414] "containerMap: removing container" podUID="865e10fe-0a6f-40bf-84c4-5f8b0153d891" containerName="payload-pre-processing" Oct 30 01:10:13.495973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.495797 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865e10fe-0a6f-40bf-84c4-5f8b0153d891" containerName="payload-pre-processing" Oct 30 01:10:13.495973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.495914 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="865e10fe-0a6f-40bf-84c4-5f8b0153d891" containerName="payload-pre-processing" Oct 30 01:10:13.495973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.495930 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6cc10e-46b7-4122-a64f-ed718babf1b8" containerName="maas-api" Oct 30 01:10:13.502363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.502335 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.523480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.521961 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-d01638-openshift-default-dockercfg-rrsfd\"" Oct 30 01:10:13.534220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.534187 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-socket\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.534374 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.534230 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwphx\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-kube-api-access-fwphx\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.534374 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.534312 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-certs\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.534490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.534461 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-data\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.534543 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.534495 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-token\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.534653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.534628 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a54ef9a-e71a-45e3-af09-92b766b0432e-istiod-ca-cert\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.534788 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.534767 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-podinfo\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.534908 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.534889 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-credential-socket\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.534967 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.534926 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-envoy\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.539139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.539112 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx"] Oct 30 01:10:13.689595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.676277 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-data\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.689955 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.689923 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-token\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.690144 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.690124 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a54ef9a-e71a-45e3-af09-92b766b0432e-istiod-ca-cert\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.690443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.690423 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-podinfo\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.710595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.690713 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-credential-socket\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.712327 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.712298 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-envoy\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.712990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.712970 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-socket\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.713436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.713418 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwphx\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-kube-api-access-fwphx\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.713604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.713587 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-certs\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.713890 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.699005 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a54ef9a-e71a-45e3-af09-92b766b0432e-istiod-ca-cert\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.731591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.713380 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-socket\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.733087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.700696 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-credential-socket\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.735382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.677232 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-data\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.735536 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.716123 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-certs\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.744653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.744625 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-podinfo\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.745522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.744984 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-envoy\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.754589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.751305 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwphx\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-kube-api-access-fwphx\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.777593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.769535 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-token\") pod \"e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:13.825591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:13.825538 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:14.272361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:14.272330 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx"] Oct 30 01:10:14.272656 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:10:14.272629 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a54ef9a_e71a_45e3_af09_92b766b0432e.slice/crio-6027aad010494efb06703c8768996f2877c37114386505840c107fc8b1671ad1 WatchSource:0}: Error finding container 6027aad010494efb06703c8768996f2877c37114386505840c107fc8b1671ad1: Status 404 returned error can't find the container with id 6027aad010494efb06703c8768996f2877c37114386505840c107fc8b1671ad1 Oct 30 01:10:14.275517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:14.275483 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:14.275641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:14.275553 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:14.275641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:14.275607 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:15.074868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:15.074828 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" event={"ID":"9a54ef9a-e71a-45e3-af09-92b766b0432e","Type":"ContainerStarted","Data":"bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f"} Oct 30 01:10:15.075260 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:15.074895 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" event={"ID":"9a54ef9a-e71a-45e3-af09-92b766b0432e","Type":"ContainerStarted","Data":"6027aad010494efb06703c8768996f2877c37114386505840c107fc8b1671ad1"} Oct 30 01:10:15.164070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:15.164009 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" podStartSLOduration=2.163992473 podStartE2EDuration="2.163992473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:15.148541841 +0000 UTC m=+2153.720468088" watchObservedRunningTime="2026-10-30 01:10:15.163992473 +0000 UTC m=+2153.735918695" Oct 30 01:10:15.826819 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:15.826783 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:15.833121 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:15.833089 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:16.083323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:16.083238 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:16.084190 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:16.084167 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:10:16.201562 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:16.201521 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" podUID="facaba60-f652-49ae-8371-8f69798f1c53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:10:17.737496 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.737469 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:10:17.807162 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.807130 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-data\") pod \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.807358 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-kube-api-access-mm9kb\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-kube-api-access-mm9kb\") pod \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.807400 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-socket\") pod \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.807431 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-credential-socket\") pod \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.807542 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-podinfo\") pod \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.807629 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-data" pod "e4656a36-4d2a-4e66-969d-7f3d7b5c9358" (UID: "e4656a36-4d2a-4e66-969d-7f3d7b5c9358"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.807853 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-credential-socket" pod "e4656a36-4d2a-4e66-969d-7f3d7b5c9358" (UID: "e4656a36-4d2a-4e66-969d-7f3d7b5c9358"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.808328 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istiod-ca-cert\") pod \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.808379 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-envoy\") pod \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.808416 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-certs\") pod \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.808443 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-socket" pod "e4656a36-4d2a-4e66-969d-7f3d7b5c9358" (UID: "e4656a36-4d2a-4e66-969d-7f3d7b5c9358"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:17.808508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.808479 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-token\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-token\") pod \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\" (UID: \"e4656a36-4d2a-4e66-969d-7f3d7b5c9358\") " Oct 30 01:10:17.809033 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.808708 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istiod-ca-cert" pod "e4656a36-4d2a-4e66-969d-7f3d7b5c9358" (UID: "e4656a36-4d2a-4e66-969d-7f3d7b5c9358"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:17.809413 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.809231 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:17.809413 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.809259 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:17.809413 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.809274 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:17.809413 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.809286 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:17.810663 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.810640 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-certs" pod "e4656a36-4d2a-4e66-969d-7f3d7b5c9358" (UID: "e4656a36-4d2a-4e66-969d-7f3d7b5c9358"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:17.819835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.819809 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-podinfo" pod "e4656a36-4d2a-4e66-969d-7f3d7b5c9358" (UID: "e4656a36-4d2a-4e66-969d-7f3d7b5c9358"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:10:17.820528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.820089 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-token" pod "e4656a36-4d2a-4e66-969d-7f3d7b5c9358" (UID: "e4656a36-4d2a-4e66-969d-7f3d7b5c9358"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:17.820528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.820176 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-kube-api-access-mm9kb" pod "e4656a36-4d2a-4e66-969d-7f3d7b5c9358" (UID: "e4656a36-4d2a-4e66-969d-7f3d7b5c9358"). InnerVolumeSpecName "kube-api-access-mm9kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:17.822094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.822070 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-envoy" pod "e4656a36-4d2a-4e66-969d-7f3d7b5c9358" (UID: "e4656a36-4d2a-4e66-969d-7f3d7b5c9358"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:17.910762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.910722 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:17.910762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.910760 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:17.910971 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.910777 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:17.910971 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.910791 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:17.910971 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:17.910810 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9kb\" (UniqueName: \"kubernetes.io/projected/e4656a36-4d2a-4e66-969d-7f3d7b5c9358-kube-api-access-mm9kb\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:18.100938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.100854 4927 generic.go:356] "Generic (PLEG): container finished" podID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerID="416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351" exitCode=0 Oct 30 01:10:18.100938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.100908 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" Oct 30 01:10:18.101116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.100942 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" event={"ID":"e4656a36-4d2a-4e66-969d-7f3d7b5c9358","Type":"ContainerDied","Data":"416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351"} Oct 30 01:10:18.101116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.101009 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms" event={"ID":"e4656a36-4d2a-4e66-969d-7f3d7b5c9358","Type":"ContainerDied","Data":"49b0b8a3d13ca053603492c57fbce96161a456a81fa2b6c931965657b41e9dc7"} Oct 30 01:10:18.101116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.101038 4927 scope.go:122] "RemoveContainer" containerID="416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351" Oct 30 01:10:18.113074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.113040 4927 scope.go:122] "RemoveContainer" containerID="416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351" Oct 30 01:10:18.113313 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:10:18.113288 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351\": container with ID starting with 416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351 not found: ID does not exist" containerID="416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351" Oct 30 01:10:18.113384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.113320 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351"} err="failed to get container status \"416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351\": rpc error: code = NotFound desc = could not find container \"416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351\": container with ID starting with 416a1b63120a5a18767d25b5a076cc82a9d17e169f2216bb0ae5a090e6701351 not found: ID does not exist" Oct 30 01:10:18.182310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.182278 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms"] Oct 30 01:10:18.191942 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.191919 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-dq5ms"] Oct 30 01:10:18.610957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.610928 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:10:18.727820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.726657 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/facaba60-f652-49ae-8371-8f69798f1c53-kube-api-access-bf4kx\" (UniqueName: \"kubernetes.io/projected/facaba60-f652-49ae-8371-8f69798f1c53-kube-api-access-bf4kx\") pod \"facaba60-f652-49ae-8371-8f69798f1c53\" (UID: \"facaba60-f652-49ae-8371-8f69798f1c53\") " Oct 30 01:10:18.727998 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.727881 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/facaba60-f652-49ae-8371-8f69798f1c53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facaba60-f652-49ae-8371-8f69798f1c53-plugins-config-volume\") pod \"facaba60-f652-49ae-8371-8f69798f1c53\" (UID: \"facaba60-f652-49ae-8371-8f69798f1c53\") " Oct 30 01:10:18.729462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.729192 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/facaba60-f652-49ae-8371-8f69798f1c53-plugins-config-volume" pod "facaba60-f652-49ae-8371-8f69798f1c53" (UID: "facaba60-f652-49ae-8371-8f69798f1c53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:18.731109 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.731082 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facaba60-f652-49ae-8371-8f69798f1c53-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:18.735554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.735510 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/facaba60-f652-49ae-8371-8f69798f1c53-kube-api-access-bf4kx" pod "facaba60-f652-49ae-8371-8f69798f1c53" (UID: "facaba60-f652-49ae-8371-8f69798f1c53"). InnerVolumeSpecName "kube-api-access-bf4kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:18.832460 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:18.832421 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf4kx\" (UniqueName: \"kubernetes.io/projected/facaba60-f652-49ae-8371-8f69798f1c53-kube-api-access-bf4kx\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:19.112195 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.112107 4927 generic.go:356] "Generic (PLEG): container finished" podID="facaba60-f652-49ae-8371-8f69798f1c53" containerID="a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57" exitCode=0 Oct 30 01:10:19.112195 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.112147 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" Oct 30 01:10:19.112384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.112151 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" event={"ID":"facaba60-f652-49ae-8371-8f69798f1c53","Type":"ContainerDied","Data":"a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57"} Oct 30 01:10:19.112384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.112267 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd" event={"ID":"facaba60-f652-49ae-8371-8f69798f1c53","Type":"ContainerDied","Data":"3a0cfc3d9eb79f2f579034bb8398c8b704a6ac0ebcb685dd86d4ba80b7dffa98"} Oct 30 01:10:19.112384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.112296 4927 scope.go:122] "RemoveContainer" containerID="a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57" Oct 30 01:10:19.124716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.124697 4927 scope.go:122] "RemoveContainer" containerID="a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57" Oct 30 01:10:19.124996 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:10:19.124980 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57\": container with ID starting with a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57 not found: ID does not exist" containerID="a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57" Oct 30 01:10:19.125068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.125009 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57"} err="failed to get container status \"a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57\": rpc error: code = NotFound desc = could not find container \"a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57\": container with ID starting with a873fc46f495363ed579393f4b66a8cacaffb29a2d226dcdbe026705dbe80a57 not found: ID does not exist" Oct 30 01:10:19.166542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.166512 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd"] Oct 30 01:10:19.174918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.174891 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1da7b2-6cbbfdc77b-nc6fd"] Oct 30 01:10:19.576875 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.576838 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" path="/var/lib/kubelet/pods/e4656a36-4d2a-4e66-969d-7f3d7b5c9358/volumes" Oct 30 01:10:19.577445 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.577422 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="facaba60-f652-49ae-8371-8f69798f1c53" path="/var/lib/kubelet/pods/facaba60-f652-49ae-8371-8f69798f1c53/volumes" Oct 30 01:10:19.806451 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.806418 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b"] Oct 30 01:10:19.807358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.807337 4927 cpu_manager.go:414] "containerMap: removing container" podUID="facaba60-f652-49ae-8371-8f69798f1c53" containerName="payload-processing" Oct 30 01:10:19.807358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.807357 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="facaba60-f652-49ae-8371-8f69798f1c53" containerName="payload-processing" Oct 30 01:10:19.807565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.807382 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerName="istio-proxy" Oct 30 01:10:19.807565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.807389 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerName="istio-proxy" Oct 30 01:10:19.807565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.807561 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="facaba60-f652-49ae-8371-8f69798f1c53" containerName="payload-processing" Oct 30 01:10:19.807565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.807593 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4656a36-4d2a-4e66-969d-7f3d7b5c9358" containerName="istio-proxy" Oct 30 01:10:19.812822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.812801 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:19.830247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.830195 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b"] Oct 30 01:10:19.852541 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.852502 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b\" (UID: \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:19.853047 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.852725 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5776\" (UniqueName: \"kubernetes.io/projected/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-kube-api-access-z5776\") pod \"payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b\" (UID: \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:19.859127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.859101 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-d01638-dockercfg-7zzt9\"" Oct 30 01:10:19.859396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:19.859380 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-d01638\"" Oct 30 01:10:20.076593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:20.007843 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5776\" (UniqueName: \"kubernetes.io/projected/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-kube-api-access-z5776\") pod \"payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b\" (UID: \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:20.084671 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:20.077126 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b\" (UID: \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:20.104678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:20.086162 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b\" (UID: \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:20.110708 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:20.110675 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5776\" (UniqueName: \"kubernetes.io/projected/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-kube-api-access-z5776\") pod \"payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b\" (UID: \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:20.148265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:20.148224 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:20.461350 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:20.461319 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b"] Oct 30 01:10:20.461876 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:10:20.461844 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35f1d3f_c7b4_46c0_b342_4d97da8aff8d.slice/crio-d62eee9e6425ebabe3d0251da6736162971955334ed5fbabea40897715dd5b88 WatchSource:0}: Error finding container d62eee9e6425ebabe3d0251da6736162971955334ed5fbabea40897715dd5b88: Status 404 returned error can't find the container with id d62eee9e6425ebabe3d0251da6736162971955334ed5fbabea40897715dd5b88 Oct 30 01:10:21.201277 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:21.201220 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" event={"ID":"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d","Type":"ContainerStarted","Data":"f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71"} Oct 30 01:10:21.201834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:21.201308 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" event={"ID":"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d","Type":"ContainerStarted","Data":"d62eee9e6425ebabe3d0251da6736162971955334ed5fbabea40897715dd5b88"} Oct 30 01:10:21.201834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:21.201345 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:21.246691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:21.246640 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" podStartSLOduration=2.246626575 podStartE2EDuration="2.246626575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:21.242554352 +0000 UTC m=+2159.814480573" watchObservedRunningTime="2026-10-30 01:10:21.246626575 +0000 UTC m=+2159.818552830" Oct 30 01:10:24.370889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.370853 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn"] Oct 30 01:10:24.389275 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.389251 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t"] Oct 30 01:10:24.391957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.389480 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.401918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.401013 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn"] Oct 30 01:10:24.401918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.401059 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t"] Oct 30 01:10:24.401918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.401156 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.470013 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.469981 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.470111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470016 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.470111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470047 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.470196 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470159 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.470196 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470187 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae00aad-022f-4c43-9680-1dd55a8cad48-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.470268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470260 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.470303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470281 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.470358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470338 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddlj\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-kube-api-access-bddlj\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.470398 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470384 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.470505 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470486 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.470546 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470534 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lv5t\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-kube-api-access-5lv5t\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.470602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470564 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.470703 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470687 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.470772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470761 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.470894 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470878 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b29146-fe62-4512-b201-b96133d2b510-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.470958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470903 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.470958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.470949 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b29146-fe62-4512-b201-b96133d2b510-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.471035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.471015 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.572386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572348 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.572386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572391 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.572697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572429 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.572697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572459 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.572697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572489 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae00aad-022f-4c43-9680-1dd55a8cad48-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.572697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572556 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.572697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572625 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.572697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572657 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddlj\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-kube-api-access-bddlj\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572736 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572834 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572863 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572896 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lv5t\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-kube-api-access-5lv5t\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.573068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572907 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.573068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.572930 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.573068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573011 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573074 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573102 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.573443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573135 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573195 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b29146-fe62-4512-b201-b96133d2b510-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.573443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573221 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.573443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573273 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b29146-fe62-4512-b201-b96133d2b510-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.573443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573314 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.573443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573344 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573407 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573453 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.573956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573461 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.573956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.573897 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae00aad-022f-4c43-9680-1dd55a8cad48-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.574253 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.574226 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b29146-fe62-4512-b201-b96133d2b510-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.575860 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.575808 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.576046 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.576020 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.576135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.576112 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.576934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.576912 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b29146-fe62-4512-b201-b96133d2b510-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.596114 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.596083 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lv5t\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-kube-api-access-5lv5t\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.596882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.596840 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.597919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.597875 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.601279 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.601214 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddlj\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-kube-api-access-bddlj\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:24.749662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.732868 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:24.749662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:24.733362 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:25.180429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.180289 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t"] Oct 30 01:10:25.180563 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:10:25.180449 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b29146_fe62_4512_b201_b96133d2b510.slice/crio-42353f893db89bc149cf00f2874a926a92a7a31ff3a4406c5b3d7b56ddaac0b4 WatchSource:0}: Error finding container 42353f893db89bc149cf00f2874a926a92a7a31ff3a4406c5b3d7b56ddaac0b4: Status 404 returned error can't find the container with id 42353f893db89bc149cf00f2874a926a92a7a31ff3a4406c5b3d7b56ddaac0b4 Oct 30 01:10:25.254041 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.253987 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:25.254181 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.254095 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:25.254181 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.254139 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:25.256458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.255959 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:25.256458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.256056 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:25.256458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.256095 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:25.257166 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.256947 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn"] Oct 30 01:10:25.366531 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.366494 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" event={"ID":"dae00aad-022f-4c43-9680-1dd55a8cad48","Type":"ContainerStarted","Data":"8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746"} Oct 30 01:10:25.366722 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.366557 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" event={"ID":"dae00aad-022f-4c43-9680-1dd55a8cad48","Type":"ContainerStarted","Data":"f4bf8bc6c9198497f5bcfc11bccf73348489bdd35ee6a2995a6a9fc1d1e1b6be"} Oct 30 01:10:25.368710 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.368683 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" event={"ID":"59b29146-fe62-4512-b201-b96133d2b510","Type":"ContainerStarted","Data":"59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43"} Oct 30 01:10:25.368841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.368733 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" event={"ID":"59b29146-fe62-4512-b201-b96133d2b510","Type":"ContainerStarted","Data":"42353f893db89bc149cf00f2874a926a92a7a31ff3a4406c5b3d7b56ddaac0b4"} Oct 30 01:10:25.411596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.411332 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" podStartSLOduration=1.411312535 podStartE2EDuration="1.411312535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:25.409356952 +0000 UTC m=+2163.981283174" watchObservedRunningTime="2026-10-30 01:10:25.411312535 +0000 UTC m=+2163.983238758" Oct 30 01:10:25.459618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.459471 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" podStartSLOduration=1.459453893 podStartE2EDuration="1.459453893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:25.454949983 +0000 UTC m=+2164.026876205" watchObservedRunningTime="2026-10-30 01:10:25.459453893 +0000 UTC m=+2164.031380114" Oct 30 01:10:25.734019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.733939 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:25.734175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:25.734029 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:26.785298 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:26.785258 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:10:26.787589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:26.787538 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:10:26.788035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:26.786394 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 01:10:26.788227 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:26.788205 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 01:10:27.734239 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:27.734198 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:10:27.734522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:27.734491 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:10:27.735103 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:27.734410 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 01:10:27.735302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:27.735272 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 01:10:28.734680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:28.734643 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:10:28.735160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:28.735024 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:10:28.735160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:28.734695 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 01:10:28.735160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:28.735096 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 01:10:29.733885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:29.733845 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 01:10:29.734105 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:29.733905 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 01:10:29.734327 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:29.734302 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:10:29.734426 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:29.734343 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:10:30.734245 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:30.734206 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 01:10:30.734686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:30.734277 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 01:10:30.734791 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:30.734763 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:10:30.734838 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:30.734817 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:10:31.642718 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:31.642427 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj"] Oct 30 01:10:31.643149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:31.643089 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" containerID="cri-o://33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc" gracePeriod=30 Oct 30 01:10:31.761542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:31.761087 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:10:31.761542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:31.761161 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:10:31.762206 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:31.762180 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 01:10:31.776589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:31.762322 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 01:10:32.294315 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:32.294281 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:10:32.942834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:32.942800 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:32.943520 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:32.943492 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:32.945248 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:32.945226 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:10:32.948697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:32.948674 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:32.949247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:32.949223 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:32.950446 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:32.950428 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:10:34.482600 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:10:34.482543 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:10:35.206788 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:35.206749 4927 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:10:35.207215 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:35.207176 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:10:37.015689 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.015656 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:10:37.105857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.105803 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-podinfo\") pod \"709ce488-5c45-4694-a7f7-bb177fa1acfe\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " Oct 30 01:10:37.106063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.105904 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709ce488-5c45-4694-a7f7-bb177fa1acfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709ce488-5c45-4694-a7f7-bb177fa1acfe-istiod-ca-cert\") pod \"709ce488-5c45-4694-a7f7-bb177fa1acfe\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " Oct 30 01:10:37.106063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.105937 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-envoy\") pod \"709ce488-5c45-4694-a7f7-bb177fa1acfe\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " Oct 30 01:10:37.106063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.106005 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-kube-api-access-chqbw\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-kube-api-access-chqbw\") pod \"709ce488-5c45-4694-a7f7-bb177fa1acfe\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " Oct 30 01:10:37.106063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.106040 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-socket\") pod \"709ce488-5c45-4694-a7f7-bb177fa1acfe\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " Oct 30 01:10:37.106351 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.106118 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-certs\") pod \"709ce488-5c45-4694-a7f7-bb177fa1acfe\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " Oct 30 01:10:37.106351 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.106200 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-token\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-token\") pod \"709ce488-5c45-4694-a7f7-bb177fa1acfe\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " Oct 30 01:10:37.106452 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.106400 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-data\") pod \"709ce488-5c45-4694-a7f7-bb177fa1acfe\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " Oct 30 01:10:37.106509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.106457 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-credential-socket\") pod \"709ce488-5c45-4694-a7f7-bb177fa1acfe\" (UID: \"709ce488-5c45-4694-a7f7-bb177fa1acfe\") " Oct 30 01:10:37.106509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.106458 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-socket" pod "709ce488-5c45-4694-a7f7-bb177fa1acfe" (UID: "709ce488-5c45-4694-a7f7-bb177fa1acfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.107030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.106997 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-certs" pod "709ce488-5c45-4694-a7f7-bb177fa1acfe" (UID: "709ce488-5c45-4694-a7f7-bb177fa1acfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.107290 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.107238 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-data" pod "709ce488-5c45-4694-a7f7-bb177fa1acfe" (UID: "709ce488-5c45-4694-a7f7-bb177fa1acfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.107290 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.107252 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-credential-socket" pod "709ce488-5c45-4694-a7f7-bb177fa1acfe" (UID: "709ce488-5c45-4694-a7f7-bb177fa1acfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.107806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.107716 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.107806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.107742 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.107806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.107755 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.107806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.107768 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.108248 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.108225 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709ce488-5c45-4694-a7f7-bb177fa1acfe-istiod-ca-cert" pod "709ce488-5c45-4694-a7f7-bb177fa1acfe" (UID: "709ce488-5c45-4694-a7f7-bb177fa1acfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:37.108781 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.108741 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-podinfo" pod "709ce488-5c45-4694-a7f7-bb177fa1acfe" (UID: "709ce488-5c45-4694-a7f7-bb177fa1acfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:10:37.109065 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.109042 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-token" pod "709ce488-5c45-4694-a7f7-bb177fa1acfe" (UID: "709ce488-5c45-4694-a7f7-bb177fa1acfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:37.109619 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.109540 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-kube-api-access-chqbw" pod "709ce488-5c45-4694-a7f7-bb177fa1acfe" (UID: "709ce488-5c45-4694-a7f7-bb177fa1acfe"). InnerVolumeSpecName "kube-api-access-chqbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:37.110475 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.110451 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-envoy" pod "709ce488-5c45-4694-a7f7-bb177fa1acfe" (UID: "709ce488-5c45-4694-a7f7-bb177fa1acfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:37.209346 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.209267 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.209346 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.209295 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.209346 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.209306 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709ce488-5c45-4694-a7f7-bb177fa1acfe-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.209346 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.209315 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709ce488-5c45-4694-a7f7-bb177fa1acfe-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.209346 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.209325 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chqbw\" (UniqueName: \"kubernetes.io/projected/709ce488-5c45-4694-a7f7-bb177fa1acfe-kube-api-access-chqbw\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:10:37.671015 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.670979 4927 generic.go:356] "Generic (PLEG): container finished" podID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerID="33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc" exitCode=0 Oct 30 01:10:37.671214 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.671044 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" Oct 30 01:10:37.671214 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.671073 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" event={"ID":"709ce488-5c45-4694-a7f7-bb177fa1acfe","Type":"ContainerDied","Data":"33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc"} Oct 30 01:10:37.671214 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.671145 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj" event={"ID":"709ce488-5c45-4694-a7f7-bb177fa1acfe","Type":"ContainerDied","Data":"37836a8563c85efca07cf90f543ba1f41348e1daa0e38d2b0a5dc99a44b284ff"} Oct 30 01:10:37.671214 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.671179 4927 scope.go:122] "RemoveContainer" containerID="33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc" Oct 30 01:10:37.686918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.686898 4927 scope.go:122] "RemoveContainer" containerID="1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b" Oct 30 01:10:37.698269 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.698237 4927 scope.go:122] "RemoveContainer" containerID="33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc" Oct 30 01:10:37.698540 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:10:37.698521 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc\": container with ID starting with 33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc not found: ID does not exist" containerID="33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc" Oct 30 01:10:37.698661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.698546 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc"} err="failed to get container status \"33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc\": rpc error: code = NotFound desc = could not find container \"33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc\": container with ID starting with 33305c4690a309ef9834b735734f5f3695b8f04c4f78281105e0daded67f09bc not found: ID does not exist" Oct 30 01:10:37.698661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.698562 4927 scope.go:122] "RemoveContainer" containerID="1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b" Oct 30 01:10:37.698865 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:10:37.698842 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b\": container with ID starting with 1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b not found: ID does not exist" containerID="1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b" Oct 30 01:10:37.698921 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.698869 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b"} err="failed to get container status \"1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b\": rpc error: code = NotFound desc = could not find container \"1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b\": container with ID starting with 1b44c9083e1600c39c78a1d89608def2ccacd243393b37367bc0cfb4e445392b not found: ID does not exist" Oct 30 01:10:37.734994 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.734964 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj"] Oct 30 01:10:37.740044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:37.740017 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-ch8zj"] Oct 30 01:10:39.016369 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.016329 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx"] Oct 30 01:10:39.017427 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.017406 4927 cpu_manager.go:414] "containerMap: removing container" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" Oct 30 01:10:39.017480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.017428 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" Oct 30 01:10:39.017480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.017447 4927 cpu_manager.go:414] "containerMap: removing container" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" Oct 30 01:10:39.017480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.017453 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" Oct 30 01:10:39.017631 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.017619 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" Oct 30 01:10:39.023018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.022995 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.049126 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.049089 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx"] Oct 30 01:10:39.145164 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.145128 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.145344 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.145171 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.145344 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.145264 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjmrm\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-kube-api-access-kjmrm\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.145439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.145365 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.145439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.145422 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.145527 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.145479 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.145610 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.145560 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.145725 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.145698 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.145764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.145722 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.247655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.247613 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.247834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.247666 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.247834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.247716 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmrm\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-kube-api-access-kjmrm\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.247834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.247814 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.247989 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.247840 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.247989 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.247893 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.248098 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.247986 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.248150 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.248130 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.248207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.248166 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.248300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.248271 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.248403 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.248381 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.248631 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.248596 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.248772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.248631 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.248921 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.248888 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.250163 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.250139 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.250745 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.250728 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.270085 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.270028 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.270219 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.270200 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmrm\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-kube-api-access-kjmrm\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.341553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.341522 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:39.573279 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.573198 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" path="/var/lib/kubelet/pods/709ce488-5c45-4694-a7f7-bb177fa1acfe/volumes" Oct 30 01:10:39.684131 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:10:39.684086 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a221756_2b9e_4fbc_a2ba_84b76f88fe89.slice/crio-beebd076ae5e76922cffc4c366bad63f771b305ce9aa9db05b7b21edbc908eee WatchSource:0}: Error finding container beebd076ae5e76922cffc4c366bad63f771b305ce9aa9db05b7b21edbc908eee: Status 404 returned error can't find the container with id beebd076ae5e76922cffc4c366bad63f771b305ce9aa9db05b7b21edbc908eee Oct 30 01:10:39.684444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.684422 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx"] Oct 30 01:10:39.686921 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.686881 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:39.687068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.686951 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:39.687068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:39.686981 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:40.698341 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:40.698303 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" event={"ID":"0a221756-2b9e-4fbc-a2ba-84b76f88fe89","Type":"ContainerStarted","Data":"b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e"} Oct 30 01:10:40.698794 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:40.698366 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" event={"ID":"0a221756-2b9e-4fbc-a2ba-84b76f88fe89","Type":"ContainerStarted","Data":"beebd076ae5e76922cffc4c366bad63f771b305ce9aa9db05b7b21edbc908eee"} Oct 30 01:10:40.753761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:40.753705 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" podStartSLOduration=2.75369022 podStartE2EDuration="2.75369022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:40.752196912 +0000 UTC m=+2179.324123157" watchObservedRunningTime="2026-10-30 01:10:40.75369022 +0000 UTC m=+2179.325616442" Oct 30 01:10:41.342697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:41.342665 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:41.344373 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:41.344342 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:10:41.344491 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:41.344399 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:10:42.342825 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:42.342786 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:10:42.343305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:42.342845 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:10:43.342302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:43.342266 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:10:43.342511 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:43.342332 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:10:44.342602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:44.342467 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:10:44.342602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:44.342535 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:10:45.342173 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:45.342133 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:10:45.342359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:45.342192 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:10:46.343209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:46.343164 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:10:46.343678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:46.343259 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:10:47.347913 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:47.347876 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:47.348383 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:47.348302 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:47.349034 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:47.349009 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:10:51.428327 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.428290 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8"] Oct 30 01:10:51.429959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.429923 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="709ce488-5c45-4694-a7f7-bb177fa1acfe" containerName="istio-proxy" Oct 30 01:10:51.435158 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.435129 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.456286 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.456254 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8"] Oct 30 01:10:51.516406 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.516378 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-bea3ef-openshift-default-dockercfg-hbbcs\"" Oct 30 01:10:51.583674 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.583639 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istiod-ca-cert\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.583674 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.583679 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66zfd\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-kube-api-access-66zfd\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.583879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.583771 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.583879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.583844 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-envoy\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.583950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.583900 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-data\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.583950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.583943 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-podinfo\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.584039 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.584026 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-token\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.584078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.584070 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-certs\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.584141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.584128 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-credential-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.686968 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-podinfo\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.687066 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-token\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.687142 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-certs\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.687185 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-credential-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.687426 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istiod-ca-cert\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.687454 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66zfd\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-kube-api-access-66zfd\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.687533 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.687663 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-envoy\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.687768 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-data\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.689702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.688299 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-data\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.694832 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.694796 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-podinfo\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.695789 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.695762 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-certs\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.696335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.696302 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-credential-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.698334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.698301 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istiod-ca-cert\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.699483 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.699452 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.710631 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.710129 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-envoy\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.729211 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.729176 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66zfd\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-kube-api-access-66zfd\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.736175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.736142 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-token\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:51.757293 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:51.757257 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:52.094306 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:52.094276 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8"] Oct 30 01:10:52.094872 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:10:52.094845 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d9aaa7_e667_4fb5_86a7_b4a0389dd87d.slice/crio-1b0fa67ca24acf973fc6be98178fb9edb92df075f37d5df09820aa16769a9afc WatchSource:0}: Error finding container 1b0fa67ca24acf973fc6be98178fb9edb92df075f37d5df09820aa16769a9afc: Status 404 returned error can't find the container with id 1b0fa67ca24acf973fc6be98178fb9edb92df075f37d5df09820aa16769a9afc Oct 30 01:10:52.097641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:52.097603 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:52.097704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:52.097677 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:52.097745 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:52.097707 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:52.900383 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:52.900161 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" event={"ID":"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d","Type":"ContainerStarted","Data":"4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4"} Oct 30 01:10:52.900383 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:52.900270 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" event={"ID":"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d","Type":"ContainerStarted","Data":"1b0fa67ca24acf973fc6be98178fb9edb92df075f37d5df09820aa16769a9afc"} Oct 30 01:10:52.954999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:52.954939 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" podStartSLOduration=1.954921629 podStartE2EDuration="1.954921629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:52.953540277 +0000 UTC m=+2191.525466498" watchObservedRunningTime="2026-10-30 01:10:52.954921629 +0000 UTC m=+2191.526847852" Oct 30 01:10:53.758949 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:53.758908 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:53.760442 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:53.760413 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 01:10:53.760592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:53.760467 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 01:10:54.758497 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:54.758450 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 01:10:54.758935 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:54.758523 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 01:10:55.758979 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:55.758947 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 01:10:55.759547 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:55.759012 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 01:10:56.764699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:56.764657 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 01:10:56.765298 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:56.765264 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 01:10:56.770946 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:56.770904 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7"] Oct 30 01:10:56.771181 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:56.771157 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" containerID="cri-o://2b0faaec7d116dad7f6568d36ab0dcb19624fd5172cc0b27cfca9bf9794b5cdc" gracePeriod=30 Oct 30 01:10:57.758664 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:57.758628 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 01:10:57.758857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:57.758696 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 01:10:58.758939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:58.758902 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 01:10:58.759469 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:58.758977 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 01:10:59.766532 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:59.766497 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:59.767870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:59.767130 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:10:59.768951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:10:59.768931 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:11:02.154554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.154516 4927 generic.go:356] "Generic (PLEG): container finished" podID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerID="2b0faaec7d116dad7f6568d36ab0dcb19624fd5172cc0b27cfca9bf9794b5cdc" exitCode=0 Oct 30 01:11:02.155108 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.154606 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" event={"ID":"0e4711fa-e187-441c-8015-0fb71f9cd69f","Type":"ContainerDied","Data":"2b0faaec7d116dad7f6568d36ab0dcb19624fd5172cc0b27cfca9bf9794b5cdc"} Oct 30 01:11:02.155108 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.154680 4927 scope.go:122] "RemoveContainer" containerID="4079a1a32d247cd9f68a51b75301b2374dd8c4ee29ceceadd10e32df6fac92f7" Oct 30 01:11:02.512428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.512395 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:11:02.601961 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.601905 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-data\") pod \"0e4711fa-e187-441c-8015-0fb71f9cd69f\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " Oct 30 01:11:02.602175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602016 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-credential-socket\") pod \"0e4711fa-e187-441c-8015-0fb71f9cd69f\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " Oct 30 01:11:02.602175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602118 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4711fa-e187-441c-8015-0fb71f9cd69f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4711fa-e187-441c-8015-0fb71f9cd69f-istiod-ca-cert\") pod \"0e4711fa-e187-441c-8015-0fb71f9cd69f\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " Oct 30 01:11:02.602175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602159 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-socket\") pod \"0e4711fa-e187-441c-8015-0fb71f9cd69f\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " Oct 30 01:11:02.602175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602160 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-data" pod "0e4711fa-e187-441c-8015-0fb71f9cd69f" (UID: "0e4711fa-e187-441c-8015-0fb71f9cd69f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:02.602421 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602256 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-credential-socket" pod "0e4711fa-e187-441c-8015-0fb71f9cd69f" (UID: "0e4711fa-e187-441c-8015-0fb71f9cd69f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:02.602421 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602283 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-envoy\") pod \"0e4711fa-e187-441c-8015-0fb71f9cd69f\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " Oct 30 01:11:02.602421 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602397 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-kube-api-access-pj94g\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-kube-api-access-pj94g\") pod \"0e4711fa-e187-441c-8015-0fb71f9cd69f\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " Oct 30 01:11:02.602607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602441 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-certs\") pod \"0e4711fa-e187-441c-8015-0fb71f9cd69f\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " Oct 30 01:11:02.602607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602471 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-podinfo\") pod \"0e4711fa-e187-441c-8015-0fb71f9cd69f\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " Oct 30 01:11:02.602607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602486 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-socket" pod "0e4711fa-e187-441c-8015-0fb71f9cd69f" (UID: "0e4711fa-e187-441c-8015-0fb71f9cd69f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:02.602607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602544 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4711fa-e187-441c-8015-0fb71f9cd69f-istiod-ca-cert" pod "0e4711fa-e187-441c-8015-0fb71f9cd69f" (UID: "0e4711fa-e187-441c-8015-0fb71f9cd69f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:11:02.602835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602612 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-token\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-token\") pod \"0e4711fa-e187-441c-8015-0fb71f9cd69f\" (UID: \"0e4711fa-e187-441c-8015-0fb71f9cd69f\") " Oct 30 01:11:02.602835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.602686 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-certs" pod "0e4711fa-e187-441c-8015-0fb71f9cd69f" (UID: "0e4711fa-e187-441c-8015-0fb71f9cd69f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:02.604144 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.603671 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:11:02.604144 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.603698 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:11:02.604144 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.603712 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4711fa-e187-441c-8015-0fb71f9cd69f-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:11:02.604144 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.603726 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:11:02.604144 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.603739 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:11:02.605360 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.605313 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-envoy" pod "0e4711fa-e187-441c-8015-0fb71f9cd69f" (UID: "0e4711fa-e187-441c-8015-0fb71f9cd69f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:02.605474 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.605432 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-kube-api-access-pj94g" pod "0e4711fa-e187-441c-8015-0fb71f9cd69f" (UID: "0e4711fa-e187-441c-8015-0fb71f9cd69f"). InnerVolumeSpecName "kube-api-access-pj94g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:02.605738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.605716 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-token" pod "0e4711fa-e187-441c-8015-0fb71f9cd69f" (UID: "0e4711fa-e187-441c-8015-0fb71f9cd69f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:02.606744 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.606719 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-podinfo" pod "0e4711fa-e187-441c-8015-0fb71f9cd69f" (UID: "0e4711fa-e187-441c-8015-0fb71f9cd69f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:11:02.705284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.705242 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:11:02.705284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.705279 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj94g\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-kube-api-access-pj94g\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:11:02.705284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.705292 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:11:02.705549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:02.705301 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4711fa-e187-441c-8015-0fb71f9cd69f-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:11:03.165509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:03.165473 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" Oct 30 01:11:03.165986 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:03.165508 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7" event={"ID":"0e4711fa-e187-441c-8015-0fb71f9cd69f","Type":"ContainerDied","Data":"292a00a67710a3b2df54d8b7b206421425546371155f2095b9a56adc2a7e3d2e"} Oct 30 01:11:03.165986 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:03.165586 4927 scope.go:122] "RemoveContainer" containerID="2b0faaec7d116dad7f6568d36ab0dcb19624fd5172cc0b27cfca9bf9794b5cdc" Oct 30 01:11:03.229732 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:03.229678 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7"] Oct 30 01:11:03.241615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:03.241589 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-bnnd7"] Oct 30 01:11:03.574234 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:03.574157 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" path="/var/lib/kubelet/pods/0e4711fa-e187-441c-8015-0fb71f9cd69f/volumes" Oct 30 01:11:04.444608 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:11:04.444556 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:11:14.989477 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:14.989443 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6"] Oct 30 01:11:14.990477 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:14.990451 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" Oct 30 01:11:14.990477 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:14.990478 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" Oct 30 01:11:14.990792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:14.990774 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" Oct 30 01:11:14.990867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:14.990799 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" Oct 30 01:11:14.996235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:14.996214 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.015111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.015080 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6"] Oct 30 01:11:15.020205 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.020174 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c4f6f\"/\"openshift-service-ca.crt\"" Oct 30 01:11:15.023877 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.023851 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c4f6f\"/\"kube-root-ca.crt\"" Oct 30 01:11:15.024022 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.023892 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c4f6f\"/\"default-dockercfg-f7q9v\"" Oct 30 01:11:15.024022 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.023910 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c4f6f\"/\"auth-test-model-6c4f6f-kserve-self-signed-certs\"" Oct 30 01:11:15.145061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.145023 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tmp-dir\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.145241 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.145177 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-model-cache\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.145294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.145281 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmhb\" (UniqueName: \"kubernetes.io/projected/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-kube-api-access-9kmhb\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.145340 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.145328 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tls-certs\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.145374 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.145367 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-dshm\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.145480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.145467 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-home\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.246835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.246742 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-model-cache\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.247023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.246837 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmhb\" (UniqueName: \"kubernetes.io/projected/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-kube-api-access-9kmhb\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.247023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.246885 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tls-certs\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.247023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.246921 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-dshm\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.247023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.246963 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-home\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.247237 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.247085 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tmp-dir\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.247301 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.247238 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-model-cache\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.247446 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.247418 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-home\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.247541 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.247506 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tmp-dir\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.249361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.249331 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-dshm\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.249686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.249667 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tls-certs\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.300643 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.297948 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmhb\" (UniqueName: \"kubernetes.io/projected/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-kube-api-access-9kmhb\") pod \"auth-test-model-6c4f6f-kserve-75d88949c-p2hb6\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.323929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.323884 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:15.787859 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:15.787797 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6"] Oct 30 01:11:15.789093 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:11:15.789071 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod160deffa_69cd_4ef7_80e8_4aa6112e0b9d.slice/crio-d9383c5f365b747adcf8fc814231d23c2a5533368069eb34eb0287c5e10699e3 WatchSource:0}: Error finding container d9383c5f365b747adcf8fc814231d23c2a5533368069eb34eb0287c5e10699e3: Status 404 returned error can't find the container with id d9383c5f365b747adcf8fc814231d23c2a5533368069eb34eb0287c5e10699e3 Oct 30 01:11:16.528074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:16.528026 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" event={"ID":"160deffa-69cd-4ef7-80e8-4aa6112e0b9d","Type":"ContainerStarted","Data":"9a3ace08c686576e00d6ae53009d414dbdc408853b33c366fe650ff6dd2993df"} Oct 30 01:11:16.528542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:16.528089 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" event={"ID":"160deffa-69cd-4ef7-80e8-4aa6112e0b9d","Type":"ContainerStarted","Data":"d9383c5f365b747adcf8fc814231d23c2a5533368069eb34eb0287c5e10699e3"} Oct 30 01:11:16.571664 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:16.571611 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" podStartSLOduration=2.571565607 podStartE2EDuration="2.571565607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:16.568644864 +0000 UTC m=+2215.140571086" watchObservedRunningTime="2026-10-30 01:11:16.571565607 +0000 UTC m=+2215.143491829" Oct 30 01:11:17.324318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:17.324287 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:24.362597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:24.360764 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 01:11:24.526323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:24.525851 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 01:11:25.325240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:25.325200 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:25.339386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:25.339355 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:25.814945 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:25.814914 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:11:32.124797 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:32.124744 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:11:32.125291 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:32.124855 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:11:34.444967 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:11:34.444925 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:11:34.910083 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:34.910046 4927 generic.go:356] "Generic (PLEG): container finished" podID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerID="bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f" exitCode=0 Oct 30 01:11:34.910303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:34.910125 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" event={"ID":"9a54ef9a-e71a-45e3-af09-92b766b0432e","Type":"ContainerDied","Data":"bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f"} Oct 30 01:11:34.910788 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:34.910768 4927 scope.go:122] "RemoveContainer" containerID="bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f" Oct 30 01:11:34.912685 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:34.912654 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:11:34.912753 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:34.912729 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:11:34.912799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:34.912762 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:11:35.920590 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:35.920547 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" event={"ID":"9a54ef9a-e71a-45e3-af09-92b766b0432e","Type":"ContainerStarted","Data":"5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d"} Oct 30 01:11:36.829988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:36.829949 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:11:36.830444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:36.830388 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:11:36.830692 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:36.830628 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:11:37.826439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:37.826404 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:11:37.826962 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:37.826925 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:11:38.826874 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:38.826830 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:11:38.827296 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:38.826893 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:11:39.826955 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:39.826920 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:11:39.827451 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:39.826993 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:11:40.773720 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.773674 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff"] Oct 30 01:11:40.798596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.775289 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" Oct 30 01:11:40.823595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.813056 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4711fa-e187-441c-8015-0fb71f9cd69f" containerName="istio-proxy" Oct 30 01:11:40.896594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.851929 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:11:40.929219 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.827083 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:11:40.929614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.929559 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:11:40.930945 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.867484 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff"] Oct 30 01:11:41.007941 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.968424 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-8e24a5\"" Oct 30 01:11:41.007941 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.968539 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-8e24a5-dockercfg-7lbg2\"" Oct 30 01:11:41.007941 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.996688 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62bm9\" (UniqueName: \"kubernetes.io/projected/61b4278f-a348-4a48-8cfa-53508163347f-kube-api-access-62bm9\") pod \"payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff\" (UID: \"61b4278f-a348-4a48-8cfa-53508163347f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:11:41.007941 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:40.996961 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b4278f-a348-4a48-8cfa-53508163347f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff\" (UID: \"61b4278f-a348-4a48-8cfa-53508163347f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:11:41.099946 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:41.099841 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b4278f-a348-4a48-8cfa-53508163347f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff\" (UID: \"61b4278f-a348-4a48-8cfa-53508163347f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:11:41.100388 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:41.100361 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62bm9\" (UniqueName: \"kubernetes.io/projected/61b4278f-a348-4a48-8cfa-53508163347f-kube-api-access-62bm9\") pod \"payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff\" (UID: \"61b4278f-a348-4a48-8cfa-53508163347f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:11:41.100750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:41.100723 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b4278f-a348-4a48-8cfa-53508163347f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff\" (UID: \"61b4278f-a348-4a48-8cfa-53508163347f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:11:41.124545 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:41.124511 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62bm9\" (UniqueName: \"kubernetes.io/projected/61b4278f-a348-4a48-8cfa-53508163347f-kube-api-access-62bm9\") pod \"payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff\" (UID: \"61b4278f-a348-4a48-8cfa-53508163347f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:11:41.308809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:41.308779 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:11:41.621136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:41.621100 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff"] Oct 30 01:11:41.621495 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:11:41.621466 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b4278f_a348_4a48_8cfa_53508163347f.slice/crio-0ed38482c186b66d579496e24b532a2ffb5097488a2383edf4c2a255d84df0f2 WatchSource:0}: Error finding container 0ed38482c186b66d579496e24b532a2ffb5097488a2383edf4c2a255d84df0f2: Status 404 returned error can't find the container with id 0ed38482c186b66d579496e24b532a2ffb5097488a2383edf4c2a255d84df0f2 Oct 30 01:11:41.831565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:41.831532 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:11:41.832080 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:41.832059 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:11:41.832747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:41.832725 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:11:42.006001 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:42.005964 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" event={"ID":"61b4278f-a348-4a48-8cfa-53508163347f","Type":"ContainerStarted","Data":"187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c"} Oct 30 01:11:42.006431 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:42.006021 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" event={"ID":"61b4278f-a348-4a48-8cfa-53508163347f","Type":"ContainerStarted","Data":"0ed38482c186b66d579496e24b532a2ffb5097488a2383edf4c2a255d84df0f2"} Oct 30 01:11:42.006431 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:42.006050 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:11:42.045986 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:42.045930 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" podStartSLOduration=2.045916103 podStartE2EDuration="2.045916103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:42.043949685 +0000 UTC m=+2240.615875910" watchObservedRunningTime="2026-10-30 01:11:42.045916103 +0000 UTC m=+2240.617842326" Oct 30 01:11:49.853285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.853246 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw"] Oct 30 01:11:49.858302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.858275 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:49.881785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.881750 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw"] Oct 30 01:11:49.883735 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.883707 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-60e8f4\"/\"openshift-service-ca.crt\"" Oct 30 01:11:49.883893 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.883875 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-60e8f4\"/\"default-dockercfg-d7g2l\"" Oct 30 01:11:49.884000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.883967 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-60e8f4\"/\"kube-root-ca.crt\"" Oct 30 01:11:49.885550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.885526 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-60e8f4\"/\"sub-test-model-60e8f4-kserve-self-signed-certs\"" Oct 30 01:11:49.945388 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.945351 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-home\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:49.945624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.945473 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3994e-d292-4fe4-ba1f-e61e80731b21-tls-certs\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:49.945731 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.945714 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9tt\" (UniqueName: \"kubernetes.io/projected/89a3994e-d292-4fe4-ba1f-e61e80731b21-kube-api-access-pz9tt\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:49.945774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.945742 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-model-cache\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:49.945967 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.945947 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-tmp-dir\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:49.946009 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:49.945998 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-dshm\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.047312 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.047267 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9tt\" (UniqueName: \"kubernetes.io/projected/89a3994e-d292-4fe4-ba1f-e61e80731b21-kube-api-access-pz9tt\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.047312 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.047311 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-model-cache\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.047538 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.047428 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-tmp-dir\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.047538 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.047462 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-dshm\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.047538 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.047522 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-home\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.047730 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.047629 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3994e-d292-4fe4-ba1f-e61e80731b21-tls-certs\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.047814 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.047788 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-model-cache\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.047871 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.047847 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-tmp-dir\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.048053 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.048031 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-home\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.049945 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.049924 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-dshm\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.050244 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.050222 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3994e-d292-4fe4-ba1f-e61e80731b21-tls-certs\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.070353 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.070318 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9tt\" (UniqueName: \"kubernetes.io/projected/89a3994e-d292-4fe4-ba1f-e61e80731b21-kube-api-access-pz9tt\") pod \"sub-test-model-60e8f4-kserve-576bc5c555-6znqw\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.211352 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.194104 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:50.590340 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:50.590304 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw"] Oct 30 01:11:50.590698 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:11:50.590672 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89a3994e_d292_4fe4_ba1f_e61e80731b21.slice/crio-44d842833fd1894e200f2d78d22f70789317ad9b5a0a72463f8169cc1dd98d8c WatchSource:0}: Error finding container 44d842833fd1894e200f2d78d22f70789317ad9b5a0a72463f8169cc1dd98d8c: Status 404 returned error can't find the container with id 44d842833fd1894e200f2d78d22f70789317ad9b5a0a72463f8169cc1dd98d8c Oct 30 01:11:51.092936 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:51.092894 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" event={"ID":"89a3994e-d292-4fe4-ba1f-e61e80731b21","Type":"ContainerStarted","Data":"91fa780c518c4d86ac42de5149d2cda3fdf1d29398eb6f5bb054f384f9213ac4"} Oct 30 01:11:51.093331 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:51.092942 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" event={"ID":"89a3994e-d292-4fe4-ba1f-e61e80731b21","Type":"ContainerStarted","Data":"44d842833fd1894e200f2d78d22f70789317ad9b5a0a72463f8169cc1dd98d8c"} Oct 30 01:11:51.163174 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:51.163119 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" podStartSLOduration=2.163105872 podStartE2EDuration="2.163105872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:51.160048644 +0000 UTC m=+2249.731974867" watchObservedRunningTime="2026-10-30 01:11:51.163105872 +0000 UTC m=+2249.735032093" Oct 30 01:11:51.195926 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:51.195894 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:11:53.017969 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:11:53.017930 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:12:00.195533 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:00.195495 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:12:00.209408 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:00.209369 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:12:01.190795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.190765 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:12:01.659769 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.659718 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2"] Oct 30 01:12:01.664683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.664653 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.680727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.680700 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-62f5d1-openshift-default-dockercfg-hprqt\"" Oct 30 01:12:01.689951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.689918 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-podinfo\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.690108 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.689973 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-credential-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.690174 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.690116 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-data\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.690277 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.690257 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-certs\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.690337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.690284 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afe368-c954-42cd-8d8b-6753ebab041d-istiod-ca-cert\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.690376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.690362 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-envoy\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.690411 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.690383 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-token\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.690449 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.690408 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjtql\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-kube-api-access-xjtql\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.690644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.690620 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.698886 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.698857 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2"] Oct 30 01:12:01.792196 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792154 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.792396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792258 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-podinfo\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.792396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792291 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-credential-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.792396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792354 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-data\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.792531 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792447 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-certs\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.792531 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792474 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afe368-c954-42cd-8d8b-6753ebab041d-istiod-ca-cert\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.792658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792529 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-envoy\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.792658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792554 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-token\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.792658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792635 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtql\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-kube-api-access-xjtql\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.792965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.792937 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.793118 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.793082 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-data\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.793190 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.793120 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-credential-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.793247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.793207 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-certs\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.793438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.793417 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afe368-c954-42cd-8d8b-6753ebab041d-istiod-ca-cert\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.794869 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.794850 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-envoy\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.794979 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.794959 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-podinfo\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.815284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.815242 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-token\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.815424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.815401 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtql\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-kube-api-access-xjtql\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:01.981146 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:01.981061 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:02.313010 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:12:02.312976 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3afe368_c954_42cd_8d8b_6753ebab041d.slice/crio-93a184eab0ace4998ce4de4e6eb07ed1c45a7582ebf05e50e88da5842e58e2f2 WatchSource:0}: Error finding container 93a184eab0ace4998ce4de4e6eb07ed1c45a7582ebf05e50e88da5842e58e2f2: Status 404 returned error can't find the container with id 93a184eab0ace4998ce4de4e6eb07ed1c45a7582ebf05e50e88da5842e58e2f2 Oct 30 01:12:02.313127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:02.313094 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2"] Oct 30 01:12:02.315970 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:02.315930 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:02.316066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:02.316005 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:02.316066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:02.316042 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:03.196224 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:03.196187 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" event={"ID":"a3afe368-c954-42cd-8d8b-6753ebab041d","Type":"ContainerStarted","Data":"44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c"} Oct 30 01:12:03.196659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:03.196251 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" event={"ID":"a3afe368-c954-42cd-8d8b-6753ebab041d","Type":"ContainerStarted","Data":"93a184eab0ace4998ce4de4e6eb07ed1c45a7582ebf05e50e88da5842e58e2f2"} Oct 30 01:12:03.258869 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:03.258801 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podStartSLOduration=2.258780337 podStartE2EDuration="2.258780337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:03.250767649 +0000 UTC m=+2261.822693871" watchObservedRunningTime="2026-10-30 01:12:03.258780337 +0000 UTC m=+2261.830706560" Oct 30 01:12:04.001615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:03.988550 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:04.001615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:03.998643 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:12:04.001615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:03.998703 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 01:12:04.467042 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:12:04.467007 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:12:04.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:04.994966 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:12:04.996596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:04.995031 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 01:12:05.986601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:05.983943 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:12:05.986601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:05.984007 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 01:12:06.994595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:06.981970 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:12:06.994595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:06.982054 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 01:12:07.984597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:07.982947 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:12:07.984597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:07.983017 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 01:12:08.998594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:08.994818 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:12:08.998594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:08.994885 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 01:12:09.986593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:09.983062 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:12:09.986593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:09.983131 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 01:12:10.598637 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:10.593686 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:12:10.811592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:10.810968 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" podUID="160deffa-69cd-4ef7-80e8-4aa6112e0b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": context deadline exceeded" Oct 30 01:12:11.197945 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:11.197900 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:11.198998 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:11.198972 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:11.201304 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:11.201279 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:12:28.322746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.322709 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn"] Oct 30 01:12:28.329040 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.329000 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.354094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.354062 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn"] Oct 30 01:12:28.358159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.358126 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-320045-openshift-default-dockercfg-cbfcp\"" Oct 30 01:12:28.381430 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.381398 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.381639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.381525 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.381639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.381622 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.381777 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.381676 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.381777 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.381761 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.381895 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.381814 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.381895 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.381849 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.382043 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.382023 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjf8\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-kube-api-access-4bjf8\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.382085 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.382057 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152f953f-039a-4f34-a0b9-e61bf335a7c8-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.483686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.483651 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152f953f-039a-4f34-a0b9-e61bf335a7c8-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.483882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.483818 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.483954 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.483900 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.483972 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484025 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484190 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484161 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484277 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484209 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484277 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484265 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484300 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484368 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152f953f-039a-4f34-a0b9-e61bf335a7c8-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484377 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484557 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484416 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484557 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484526 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjf8\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-kube-api-access-4bjf8\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.484685 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.484636 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.486557 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.486530 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.486799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.486779 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.506799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.506773 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.506939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.506915 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjf8\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-kube-api-access-4bjf8\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.648796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.648760 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:28.982640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.982538 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn"] Oct 30 01:12:28.989206 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:12:28.989171 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152f953f_039a_4f34_a0b9_e61bf335a7c8.slice/crio-980e362085f2a6e70618fbb104d5f9efd761905a9b7b3927cd1fc91e6ffb8e33 WatchSource:0}: Error finding container 980e362085f2a6e70618fbb104d5f9efd761905a9b7b3927cd1fc91e6ffb8e33: Status 404 returned error can't find the container with id 980e362085f2a6e70618fbb104d5f9efd761905a9b7b3927cd1fc91e6ffb8e33 Oct 30 01:12:28.992843 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.992801 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:28.992971 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.992883 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:28.992971 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:28.992921 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:29.760105 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:29.760072 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" event={"ID":"152f953f-039a-4f34-a0b9-e61bf335a7c8","Type":"ContainerStarted","Data":"1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445"} Oct 30 01:12:29.760516 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:29.760123 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" event={"ID":"152f953f-039a-4f34-a0b9-e61bf335a7c8","Type":"ContainerStarted","Data":"980e362085f2a6e70618fbb104d5f9efd761905a9b7b3927cd1fc91e6ffb8e33"} Oct 30 01:12:29.817740 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:29.816518 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" podStartSLOduration=1.816501377 podStartE2EDuration="1.816501377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:29.815214607 +0000 UTC m=+2288.387140823" watchObservedRunningTime="2026-10-30 01:12:29.816501377 +0000 UTC m=+2288.388427599" Oct 30 01:12:30.649647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:30.649614 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:30.650985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:30.650959 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:12:30.651113 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:30.651008 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 01:12:31.649566 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:31.649528 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:12:31.650211 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:31.649603 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 01:12:32.649919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:32.649834 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:12:32.649919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:32.649888 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 01:12:33.649627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:33.649557 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:12:33.649817 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:33.649649 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 01:12:34.443505 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:12:34.443466 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:12:34.649191 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:34.649148 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:12:34.649384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:34.649235 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 01:12:35.654549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:35.654515 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:35.654977 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:35.654961 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:35.655682 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:35.655655 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:12:45.875993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.875954 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc"] Oct 30 01:12:45.881510 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.881485 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:45.898792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.898769 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6c4f6f-openshift-default-dockercfg-gp89t\"" Oct 30 01:12:45.915167 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.915134 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc"] Oct 30 01:12:45.984032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.983992 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-istio-token\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:45.984202 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.984131 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-credential-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:45.984250 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.984200 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gh2\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-kube-api-access-m9gh2\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:45.984286 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.984270 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-certs\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:45.984361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.984349 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e312d-afd1-4032-909f-aa766af53204-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:45.984398 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.984385 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-envoy\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:45.984439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.984419 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e312d-afd1-4032-909f-aa766af53204-istio-podinfo\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:45.984523 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.984506 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-data\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:45.984559 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:45.984550 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.085916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.085874 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-credential-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.085916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.085923 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gh2\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-kube-api-access-m9gh2\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.086189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.085985 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-certs\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.086189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.086042 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e312d-afd1-4032-909f-aa766af53204-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.086189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.086061 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-envoy\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.086189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.086091 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e312d-afd1-4032-909f-aa766af53204-istio-podinfo\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.086189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.086166 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-data\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.086462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.086212 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.086462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.086337 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-istio-token\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.086462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.086447 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-credential-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.086822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.086796 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-data\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.087023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.086974 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e312d-afd1-4032-909f-aa766af53204-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.087087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.087033 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.087266 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.087243 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-certs\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.089235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.089212 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-envoy\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.089593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.089555 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e312d-afd1-4032-909f-aa766af53204-istio-podinfo\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.108340 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.108312 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-istio-token\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.108563 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.108541 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gh2\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-kube-api-access-m9gh2\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.198289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.198210 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:46.527510 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:12:46.527480 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667e312d_afd1_4032_909f_aa766af53204.slice/crio-7155aa3ed96f26e647582fcb104ac5e3091047ccd5efd79d3fc8722a485ee302 WatchSource:0}: Error finding container 7155aa3ed96f26e647582fcb104ac5e3091047ccd5efd79d3fc8722a485ee302: Status 404 returned error can't find the container with id 7155aa3ed96f26e647582fcb104ac5e3091047ccd5efd79d3fc8722a485ee302 Oct 30 01:12:46.528032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.528006 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc"] Oct 30 01:12:46.530750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.530716 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:46.530848 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.530784 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:46.530848 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.530814 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:46.961534 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.961495 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" event={"ID":"667e312d-afd1-4032-909f-aa766af53204","Type":"ContainerStarted","Data":"37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0"} Oct 30 01:12:46.962007 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:46.961565 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" event={"ID":"667e312d-afd1-4032-909f-aa766af53204","Type":"ContainerStarted","Data":"7155aa3ed96f26e647582fcb104ac5e3091047ccd5efd79d3fc8722a485ee302"} Oct 30 01:12:47.029608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:47.028732 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" podStartSLOduration=2.028711498 podStartE2EDuration="2.028711498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:47.027055236 +0000 UTC m=+2305.598981459" watchObservedRunningTime="2026-10-30 01:12:47.028711498 +0000 UTC m=+2305.600637721" Oct 30 01:12:47.198393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:47.198362 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:47.200458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:47.200427 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:12:47.200620 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:47.200477 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:12:48.199255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:48.199199 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:12:48.199840 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:48.199311 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:12:49.199087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:49.199045 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:12:49.199327 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:49.199124 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:12:50.202948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:50.202907 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:12:50.203586 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:50.203538 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:12:51.198949 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:51.198907 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:12:51.199224 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:51.199193 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:12:52.200271 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:52.200231 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:12:52.200984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:52.200948 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:12:53.201987 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:53.201948 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 01:12:53.202559 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:53.202524 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 01:12:54.237925 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:54.232302 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:54.237925 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:54.236493 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:54.239986 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:54.239528 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:12:58.275768 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:58.275711 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" podUID="89a3994e-d292-4fe4-ba1f-e61e80731b21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.179:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:12:58.372436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:12:58.372385 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:13:04.445266 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:13:04.445228 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:13:13.489712 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.489666 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs"] Oct 30 01:13:13.495767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.495741 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.518888 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.518852 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs"] Oct 30 01:13:13.596982 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.596944 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96669c7-c61d-464d-af50-6038a619fa07-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.597164 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.597054 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.597164 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.597077 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll54r\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-kube-api-access-ll54r\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.597251 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.597197 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.597335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.597309 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.597391 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.597378 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.597449 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.597421 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96669c7-c61d-464d-af50-6038a619fa07-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.597518 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.597484 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.597646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.597631 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699145 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699114 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699160 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96669c7-c61d-464d-af50-6038a619fa07-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699254 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699293 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll54r\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-kube-api-access-ll54r\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699479 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699445 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699610 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699582 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699667 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699686 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96669c7-c61d-464d-af50-6038a619fa07-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699715 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.699767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699748 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.700023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.699827 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.700087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.700069 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96669c7-c61d-464d-af50-6038a619fa07-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.700160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.700135 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.700310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.700150 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.702218 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.702196 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.702456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.702434 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96669c7-c61d-464d-af50-6038a619fa07-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.721544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.721520 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.721811 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.721790 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll54r\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-kube-api-access-ll54r\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:13.835341 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:13.835246 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:14.166626 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:14.166592 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs"] Oct 30 01:13:14.169981 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:14.169942 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:14.170107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:14.170020 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:14.170107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:14.170055 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:14.523766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:14.523668 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" event={"ID":"d96669c7-c61d-464d-af50-6038a619fa07","Type":"ContainerStarted","Data":"b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e"} Oct 30 01:13:14.523766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:14.523742 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" event={"ID":"d96669c7-c61d-464d-af50-6038a619fa07","Type":"ContainerStarted","Data":"1f6cfca6c48286c0f709899f0dc8f72f74e7ad2372521c93a502a1c500a283a9"} Oct 30 01:13:14.568104 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:14.568042 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" podStartSLOduration=1.568024548 podStartE2EDuration="1.568024548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:14.565959793 +0000 UTC m=+2333.137886016" watchObservedRunningTime="2026-10-30 01:13:14.568024548 +0000 UTC m=+2333.139950771" Oct 30 01:13:14.835400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:14.835308 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:15.837442 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:15.837407 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:13:15.837885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:15.837458 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" podUID="d96669c7-c61d-464d-af50-6038a619fa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 01:13:16.852592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:16.843100 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:13:16.852592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:16.843182 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" podUID="d96669c7-c61d-464d-af50-6038a619fa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 01:13:17.836027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:17.835983 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:13:17.836232 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:17.836056 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" podUID="d96669c7-c61d-464d-af50-6038a619fa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 01:13:18.836365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:18.836328 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:13:18.836810 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:18.836458 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" podUID="d96669c7-c61d-464d-af50-6038a619fa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 01:13:19.835720 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:19.835685 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:13:19.835921 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:19.835742 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" podUID="d96669c7-c61d-464d-af50-6038a619fa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 01:13:20.841756 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:20.841722 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:20.842263 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:20.842233 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:20.843047 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:20.843024 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:13:25.440688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:25.440628 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8"] Oct 30 01:13:32.692249 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.692210 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n"] Oct 30 01:13:32.699183 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.699149 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.718235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.718200 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n"] Oct 30 01:13:32.722064 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.722037 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-16c1bb\"/\"openshift-service-ca.crt\"" Oct 30 01:13:32.724513 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.722093 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-16c1bb\"/\"default-dockercfg-pv7bm\"" Oct 30 01:13:32.727157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.727134 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-16c1bb\"/\"kube-root-ca.crt\"" Oct 30 01:13:32.727286 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.727248 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-16c1bb\"/\"sub-test-model-16c1bb-kserve-self-signed-certs\"" Oct 30 01:13:32.840706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.840673 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-model-cache\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.840895 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.840713 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tls-certs\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.840895 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.840797 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tmp-dir\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.840991 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.840897 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-dshm\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.840991 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.840974 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-home\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.841085 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.841011 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28cq8\" (UniqueName: \"kubernetes.io/projected/7524c2d1-a405-4f4b-90ff-ffd280dd1840-kube-api-access-28cq8\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.943077 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.942972 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-model-cache\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.944133 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.944108 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tls-certs\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.951268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.951229 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tmp-dir\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.957590 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.957533 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-dshm\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.967801 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.958355 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-home\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.968118 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.968090 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28cq8\" (UniqueName: \"kubernetes.io/projected/7524c2d1-a405-4f4b-90ff-ffd280dd1840-kube-api-access-28cq8\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.968718 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.943755 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-model-cache\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.968889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.959232 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-home\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.969022 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.951828 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tmp-dir\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:32.987642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:32.987546 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tls-certs\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:33.014146 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:33.014111 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-dshm\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:33.044479 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:33.044443 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28cq8\" (UniqueName: \"kubernetes.io/projected/7524c2d1-a405-4f4b-90ff-ffd280dd1840-kube-api-access-28cq8\") pod \"sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:33.329236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:33.329147 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:33.643845 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:13:33.643802 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7524c2d1_a405_4f4b_90ff_ffd280dd1840.slice/crio-eaf5a829ddb57a167577db620ec31d1126554aa48704c4b08e8bb93e4b74a82c WatchSource:0}: Error finding container eaf5a829ddb57a167577db620ec31d1126554aa48704c4b08e8bb93e4b74a82c: Status 404 returned error can't find the container with id eaf5a829ddb57a167577db620ec31d1126554aa48704c4b08e8bb93e4b74a82c Oct 30 01:13:33.644134 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:33.644111 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n"] Oct 30 01:13:33.842774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:33.842731 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" event={"ID":"7524c2d1-a405-4f4b-90ff-ffd280dd1840","Type":"ContainerStarted","Data":"fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714"} Oct 30 01:13:33.843217 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:33.842788 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" event={"ID":"7524c2d1-a405-4f4b-90ff-ffd280dd1840","Type":"ContainerStarted","Data":"eaf5a829ddb57a167577db620ec31d1126554aa48704c4b08e8bb93e4b74a82c"} Oct 30 01:13:33.887325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:33.887274 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" podStartSLOduration=1.887256182 podStartE2EDuration="1.887256182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:33.88218345 +0000 UTC m=+2352.454109672" watchObservedRunningTime="2026-10-30 01:13:33.887256182 +0000 UTC m=+2352.459182405" Oct 30 01:13:34.329289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:34.329255 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:34.445954 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:13:34.445911 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:13:37.101450 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.101409 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv"] Oct 30 01:13:37.106504 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.106484 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.121161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.121137 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-33e759-openshift-default-dockercfg-6b9m2\"" Oct 30 01:13:37.140169 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.140131 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv"] Oct 30 01:13:37.199488 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.199448 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-data\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.199703 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.199553 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebc161-dea3-44cb-845b-8531f0399603-istio-podinfo\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.199703 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.199635 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.199703 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.199676 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-certs\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.199846 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.199756 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-envoy\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.199911 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.199898 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-istio-token\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.199953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.199925 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebc161-dea3-44cb-845b-8531f0399603-istiod-ca-cert\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.200031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.200017 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6tj\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-kube-api-access-mj6tj\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.200072 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.200061 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-credential-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301280 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301242 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-istio-token\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301484 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301300 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebc161-dea3-44cb-845b-8531f0399603-istiod-ca-cert\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301484 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301380 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6tj\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-kube-api-access-mj6tj\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301484 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301424 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-credential-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301515 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-data\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301554 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebc161-dea3-44cb-845b-8531f0399603-istio-podinfo\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301667 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301732 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-certs\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301823 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-envoy\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301981 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301917 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-credential-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.301981 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.301962 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-data\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.302085 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.302027 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.302209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.302181 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebc161-dea3-44cb-845b-8531f0399603-istiod-ca-cert\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.302324 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.302301 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-certs\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.304094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.304073 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-envoy\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.304292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.304271 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebc161-dea3-44cb-845b-8531f0399603-istio-podinfo\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.329030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.328999 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-istio-token\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.329332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.329311 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6tj\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-kube-api-access-mj6tj\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.424511 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.424477 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:37.759652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.759621 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv"] Oct 30 01:13:37.759826 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:13:37.759802 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ebc161_dea3_44cb_845b_8531f0399603.slice/crio-caedd7c80388e71a71c4d8dc25708f3cc0b2c6beb4a517321a9d06ae9762745c WatchSource:0}: Error finding container caedd7c80388e71a71c4d8dc25708f3cc0b2c6beb4a517321a9d06ae9762745c: Status 404 returned error can't find the container with id caedd7c80388e71a71c4d8dc25708f3cc0b2c6beb4a517321a9d06ae9762745c Oct 30 01:13:37.763038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.762998 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:37.763123 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.763082 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:37.763161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.763152 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:37.885288 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.885241 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" event={"ID":"83ebc161-dea3-44cb-845b-8531f0399603","Type":"ContainerStarted","Data":"179ee5373677eff984a1961d70f2ae2d9355ea8fc46748399af0875eea4111e8"} Oct 30 01:13:37.885482 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.885316 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" event={"ID":"83ebc161-dea3-44cb-845b-8531f0399603","Type":"ContainerStarted","Data":"caedd7c80388e71a71c4d8dc25708f3cc0b2c6beb4a517321a9d06ae9762745c"} Oct 30 01:13:37.933624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:37.933545 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podStartSLOduration=0.933525307 podStartE2EDuration="933.525307ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:37.927642468 +0000 UTC m=+2356.499568689" watchObservedRunningTime="2026-10-30 01:13:37.933525307 +0000 UTC m=+2356.505451533" Oct 30 01:13:38.437593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:38.434547 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:38.437593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:38.436359 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 01:13:38.437593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:38.436428 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 01:13:39.440005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:39.439967 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 01:13:39.440666 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:39.440611 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 01:13:40.425468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.425428 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 01:13:40.425686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.425509 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 01:13:40.500200 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.500153 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" podUID="ae44dd37-5d61-4662-a6cb-6cbc7b369d58" containerName="main" containerID="cri-o://f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c" gracePeriod=45 Oct 30 01:13:40.858961 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.858924 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:13:40.918152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.918109 4927 generic.go:356] "Generic (PLEG): container finished" podID="ae44dd37-5d61-4662-a6cb-6cbc7b369d58" containerID="f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c" exitCode=0 Oct 30 01:13:40.918345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.918168 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" Oct 30 01:13:40.918345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.918198 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" event={"ID":"ae44dd37-5d61-4662-a6cb-6cbc7b369d58","Type":"ContainerDied","Data":"f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c"} Oct 30 01:13:40.918345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.918255 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8" event={"ID":"ae44dd37-5d61-4662-a6cb-6cbc7b369d58","Type":"ContainerDied","Data":"f920b05826d91ab3db54c871ad9a0ffba42de3e6b4a10ff04c35420fea4bb31a"} Oct 30 01:13:40.918345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.918282 4927 scope.go:122] "RemoveContainer" containerID="f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c" Oct 30 01:13:40.925249 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.925217 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-kube-api-access-gwccc\" (UniqueName: \"kubernetes.io/projected/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-kube-api-access-gwccc\") pod \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " Oct 30 01:13:40.925440 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.925292 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-dshm\") pod \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " Oct 30 01:13:40.925440 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.925347 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tls-certs\") pod \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " Oct 30 01:13:40.925795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.925769 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tmp-dir\") pod \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " Oct 30 01:13:40.925898 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.925816 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-model-cache\") pod \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " Oct 30 01:13:40.925898 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.925883 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-home\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-home\") pod \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\" (UID: \"ae44dd37-5d61-4662-a6cb-6cbc7b369d58\") " Oct 30 01:13:40.926149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.926113 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tmp-dir" pod "ae44dd37-5d61-4662-a6cb-6cbc7b369d58" (UID: "ae44dd37-5d61-4662-a6cb-6cbc7b369d58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:40.926232 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.926199 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-model-cache" pod "ae44dd37-5d61-4662-a6cb-6cbc7b369d58" (UID: "ae44dd37-5d61-4662-a6cb-6cbc7b369d58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:40.926362 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.926341 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-home" pod "ae44dd37-5d61-4662-a6cb-6cbc7b369d58" (UID: "ae44dd37-5d61-4662-a6cb-6cbc7b369d58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:40.926886 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.926859 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:13:40.926992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.926890 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:13:40.926992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.926903 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:13:40.929159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.928477 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-kube-api-access-gwccc" pod "ae44dd37-5d61-4662-a6cb-6cbc7b369d58" (UID: "ae44dd37-5d61-4662-a6cb-6cbc7b369d58"). InnerVolumeSpecName "kube-api-access-gwccc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:40.929370 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.929336 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tls-certs" pod "ae44dd37-5d61-4662-a6cb-6cbc7b369d58" (UID: "ae44dd37-5d61-4662-a6cb-6cbc7b369d58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:40.929834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.929811 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-dshm" pod "ae44dd37-5d61-4662-a6cb-6cbc7b369d58" (UID: "ae44dd37-5d61-4662-a6cb-6cbc7b369d58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:40.931951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.931929 4927 scope.go:122] "RemoveContainer" containerID="f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c" Oct 30 01:13:40.932357 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:13:40.932325 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c\": container with ID starting with f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c not found: ID does not exist" containerID="f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c" Oct 30 01:13:40.932455 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:40.932365 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c"} err="failed to get container status \"f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c\": rpc error: code = NotFound desc = could not find container \"f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c\": container with ID starting with f8f38dcdaa73fd9cea44b80464c3ac5988ab3cbd5ce1ad0657df393cdd41521c not found: ID does not exist" Oct 30 01:13:41.028922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:41.028837 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:13:41.028922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:41.028869 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:13:41.028922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:41.028884 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwccc\" (UniqueName: \"kubernetes.io/projected/ae44dd37-5d61-4662-a6cb-6cbc7b369d58-kube-api-access-gwccc\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:13:41.264753 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:41.264718 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8"] Oct 30 01:13:41.301053 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:41.300837 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bea3ef/auth-test-model-bea3ef-kserve-85db74d9d9-2hgk8"] Oct 30 01:13:41.425822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:41.425783 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 01:13:41.426019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:41.425849 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 01:13:41.784595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:41.764331 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae44dd37-5d61-4662-a6cb-6cbc7b369d58" path="/var/lib/kubelet/pods/ae44dd37-5d61-4662-a6cb-6cbc7b369d58/volumes" Oct 30 01:13:42.428598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:42.427986 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 01:13:42.428598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:42.428050 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 01:13:43.330046 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:43.330010 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:43.342511 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:43.342485 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:43.425617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:43.425553 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 01:13:43.425803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:43.425647 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 01:13:44.039387 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:44.039357 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:13:44.428076 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:44.428036 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 01:13:44.428766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:44.428726 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 01:13:45.427838 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:45.427797 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 01:13:45.428143 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:45.428109 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 01:13:46.439003 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:46.438967 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:46.440598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:46.440522 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:46.448842 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:46.448815 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:13:46.971164 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:46.971123 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg"] Oct 30 01:13:46.972429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:46.972399 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ae44dd37-5d61-4662-a6cb-6cbc7b369d58" containerName="main" Oct 30 01:13:46.972429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:46.972427 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae44dd37-5d61-4662-a6cb-6cbc7b369d58" containerName="main" Oct 30 01:13:46.972656 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:46.972589 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae44dd37-5d61-4662-a6cb-6cbc7b369d58" containerName="main" Oct 30 01:13:46.982868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:46.982836 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:13:47.004892 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.004854 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg"] Oct 30 01:13:47.011926 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.011897 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f91e87\"" Oct 30 01:13:47.126315 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.112711 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmk2s\" (UniqueName: \"kubernetes.io/projected/37b181ff-a407-4f26-9d90-63131a047c6f-kube-api-access-mmk2s\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg\" (UID: \"37b181ff-a407-4f26-9d90-63131a047c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:13:47.153381 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.153334 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b181ff-a407-4f26-9d90-63131a047c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg\" (UID: \"37b181ff-a407-4f26-9d90-63131a047c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:13:47.258129 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.258028 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmk2s\" (UniqueName: \"kubernetes.io/projected/37b181ff-a407-4f26-9d90-63131a047c6f-kube-api-access-mmk2s\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg\" (UID: \"37b181ff-a407-4f26-9d90-63131a047c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:13:47.258507 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.258466 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b181ff-a407-4f26-9d90-63131a047c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg\" (UID: \"37b181ff-a407-4f26-9d90-63131a047c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:13:47.274593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.270597 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b181ff-a407-4f26-9d90-63131a047c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg\" (UID: \"37b181ff-a407-4f26-9d90-63131a047c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:13:47.333267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.333230 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmk2s\" (UniqueName: \"kubernetes.io/projected/37b181ff-a407-4f26-9d90-63131a047c6f-kube-api-access-mmk2s\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg\" (UID: \"37b181ff-a407-4f26-9d90-63131a047c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:13:47.618271 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.618182 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:13:47.929266 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:13:47.929229 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b181ff_a407_4f26_9d90_63131a047c6f.slice/crio-1b9367f3ed0ef7b5e098d9f923e298c34122fd635b4edb5748e413e6cc9cddbb WatchSource:0}: Error finding container 1b9367f3ed0ef7b5e098d9f923e298c34122fd635b4edb5748e413e6cc9cddbb: Status 404 returned error can't find the container with id 1b9367f3ed0ef7b5e098d9f923e298c34122fd635b4edb5748e413e6cc9cddbb Oct 30 01:13:47.929475 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:47.929451 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg"] Oct 30 01:13:48.069770 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:48.069731 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" event={"ID":"37b181ff-a407-4f26-9d90-63131a047c6f","Type":"ContainerStarted","Data":"4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685"} Oct 30 01:13:48.069968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:48.069800 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:13:48.069968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:48.069830 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" event={"ID":"37b181ff-a407-4f26-9d90-63131a047c6f","Type":"ContainerStarted","Data":"1b9367f3ed0ef7b5e098d9f923e298c34122fd635b4edb5748e413e6cc9cddbb"} Oct 30 01:13:48.113518 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:48.113458 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" podStartSLOduration=2.113441823 podStartE2EDuration="2.113441823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:48.109926566 +0000 UTC m=+2366.681852788" watchObservedRunningTime="2026-10-30 01:13:48.113441823 +0000 UTC m=+2366.685368098" Oct 30 01:13:59.085741 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:13:59.085709 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:14:04.444667 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:14:04.444566 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:14:22.430430 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:22.430398 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:14:22.431210 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:22.431188 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:14:23.614048 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:23.614008 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc"] Oct 30 01:14:23.620488 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:23.620442 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:23.640807 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:23.640724 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f91e87-serving-cert\"" Oct 30 01:14:23.644423 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:23.644383 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc"] Oct 30 01:14:23.842597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:23.841644 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-metrics-certs\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:23.842597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:23.841920 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52ae8-27a3-46a3-9bd0-509b6397a318-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:23.842597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:23.841992 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q678t\" (UniqueName: \"kubernetes.io/projected/74f52ae8-27a3-46a3-9bd0-509b6397a318-kube-api-access-q678t\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:23.842597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:23.842217 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.098587 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.084224 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.098587 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.084318 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-metrics-certs\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.098587 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.084622 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52ae8-27a3-46a3-9bd0-509b6397a318-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.098587 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.084697 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q678t\" (UniqueName: \"kubernetes.io/projected/74f52ae8-27a3-46a3-9bd0-509b6397a318-kube-api-access-q678t\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.104599 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.103174 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52ae8-27a3-46a3-9bd0-509b6397a318-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.104599 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:14:24.103334 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-serving-cert: secret "maas-api-e2e-ipp-a-f91e87-serving-cert" not found Oct 30 01:14:24.104599 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:14:24.103409 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls podName:74f52ae8-27a3-46a3-9bd0-509b6397a318 nodeName:}" failed. No retries permitted until 2026-10-30 01:14:24.603388936 +0000 UTC m=+2403.175315149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls") pod "maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" (UID: "74f52ae8-27a3-46a3-9bd0-509b6397a318") : secret "maas-api-e2e-ipp-a-f91e87-serving-cert" not found Oct 30 01:14:24.363534 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.363452 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-metrics-certs\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.365485 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.365453 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q678t\" (UniqueName: \"kubernetes.io/projected/74f52ae8-27a3-46a3-9bd0-509b6397a318-kube-api-access-q678t\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.656111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.656062 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.887338 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.887304 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:24.904675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:24.904634 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:25.040971 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:25.040924 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:25.046593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:25.043328 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 01:14:25.047703 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:25.047653 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" podUID="7524c2d1-a405-4f4b-90ff-ffd280dd1840" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": context deadline exceeded" Oct 30 01:14:25.264366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:25.264327 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 01:14:25.265050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:25.264947 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:25.364716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:25.364668 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 01:14:25.810745 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:25.810692 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" podUID="160deffa-69cd-4ef7-80e8-4aa6112e0b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.053973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:26.053923 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" podUID="7524c2d1-a405-4f4b-90ff-ffd280dd1840" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.054663 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:26.054639 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:14:26.081303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:26.054658 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.081303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:26.057147 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 01:14:26.174647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:26.107888 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:26.174647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:26.108312 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 01:14:26.174647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:26.135490 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 01:14:26.191724 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:26.191206 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:14:26.260623 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:26.200774 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" podUID="89a3994e-d292-4fe4-ba1f-e61e80731b21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.179:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:27.110592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:27.108037 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" Oct 30 01:14:27.190037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:27.189953 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:14:27.341768 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:27.341408 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" podUID="89a3994e-d292-4fe4-ba1f-e61e80731b21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.179:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:14:27.351005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:27.342068 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:14:27.387475 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:27.387436 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc"] Oct 30 01:14:27.390624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:27.390455 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:14:28.142352 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:28.142310 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" event={"ID":"74f52ae8-27a3-46a3-9bd0-509b6397a318","Type":"ContainerStarted","Data":"ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25"} Oct 30 01:14:28.142932 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:28.142376 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" event={"ID":"74f52ae8-27a3-46a3-9bd0-509b6397a318","Type":"ContainerStarted","Data":"db1b28f662f1220f0fc95183bd99f06fa228a081c08a2fcb8d33e400823aa179"} Oct 30 01:14:28.142932 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:28.142435 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:28.188787 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:28.188719 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" podStartSLOduration=4.721073202 podStartE2EDuration="5.188703635s" totalImagesPullingTime="467.630433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:28.183395291 +0000 UTC m=+2406.755321508" watchObservedRunningTime="2026-10-30 01:14:28.188703635 +0000 UTC m=+2406.760629857" Oct 30 01:14:34.159473 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:34.159436 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:14:34.506260 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:14:34.506175 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:14:51.606919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.606878 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8"] Oct 30 01:14:51.613743 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.613713 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.635525 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.635493 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8"] Oct 30 01:14:51.637788 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.637758 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-abf8ae-openshift-default-dockercfg-hzplz\"" Oct 30 01:14:51.744594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.742424 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-credential-socket\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.776589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.742487 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-podinfo\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.776589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.773000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-socket\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.776589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.773040 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-envoy\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.776589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.773067 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p4cm\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-kube-api-access-2p4cm\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.776589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.773139 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istiod-ca-cert\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.776589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.773188 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-token\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.776589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.773272 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-data\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.776589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.773304 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-certs\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.884594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.878060 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-socket\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.884594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.878122 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-envoy\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.884594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.878152 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p4cm\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-kube-api-access-2p4cm\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.884594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.878213 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istiod-ca-cert\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.884594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.878278 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-token\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.884594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.878356 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-data\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.884594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.878387 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-certs\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.884594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.878710 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-credential-socket\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.884594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.878756 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-podinfo\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.896592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.895393 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-data\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.896592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.896417 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-credential-socket\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.907606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.898963 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istiod-ca-cert\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.926592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.921995 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-certs\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.926592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.922510 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-socket\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.931598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.931087 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-podinfo\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.948377 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.948306 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p4cm\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-kube-api-access-2p4cm\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.970523 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.970452 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-envoy\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:51.973602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:51.972380 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-token\") pod \"e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:52.254645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:52.245116 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:52.617122 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:14:52.617090 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e21533_1c44_41fa_82a3_f5ed8296bdb4.slice/crio-9c25e38f9d3b68633b0dec6a946f030ecf841005c5ce27e1de0ab272977b0cf1 WatchSource:0}: Error finding container 9c25e38f9d3b68633b0dec6a946f030ecf841005c5ce27e1de0ab272977b0cf1: Status 404 returned error can't find the container with id 9c25e38f9d3b68633b0dec6a946f030ecf841005c5ce27e1de0ab272977b0cf1 Oct 30 01:14:52.617593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:52.617343 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8"] Oct 30 01:14:52.620278 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:52.620241 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:14:52.620398 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:52.620325 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:14:52.620398 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:52.620355 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:14:53.408468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:53.408434 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" event={"ID":"28e21533-1c44-41fa-82a3-f5ed8296bdb4","Type":"ContainerStarted","Data":"7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74"} Oct 30 01:14:53.408753 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:53.408510 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" event={"ID":"28e21533-1c44-41fa-82a3-f5ed8296bdb4","Type":"ContainerStarted","Data":"9c25e38f9d3b68633b0dec6a946f030ecf841005c5ce27e1de0ab272977b0cf1"} Oct 30 01:14:53.503475 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:53.503020 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" podStartSLOduration=2.50299368 podStartE2EDuration="2.50299368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:53.466128544 +0000 UTC m=+2432.038054769" watchObservedRunningTime="2026-10-30 01:14:53.50299368 +0000 UTC m=+2432.074919930" Oct 30 01:14:54.246588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:54.246534 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:54.252967 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:54.252937 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:54.417414 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:54.417379 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:54.418289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:54.418265 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:14:55.651450 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:55.651412 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92"] Oct 30 01:14:55.651937 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:14:55.651683 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" containerID="cri-o://736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029" gracePeriod=30 Oct 30 01:15:00.221056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.217396 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:15:00.221056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.217496 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:15:00.250103 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.250064 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2"] Oct 30 01:15:00.257583 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.257542 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:00.258111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.258077 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7"] Oct 30 01:15:00.266326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.266296 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:00.275007 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.272542 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9"] Oct 30 01:15:00.284405 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.284385 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2"] Oct 30 01:15:00.284639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.284538 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:00.307892 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.307855 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9"] Oct 30 01:15:00.336201 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.336162 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7"] Oct 30 01:15:00.346179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.346154 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:15:00.426365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.426329 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsfl\" (UniqueName: \"kubernetes.io/projected/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-kube-api-access-jnsfl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2\" (UID: \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:00.426530 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.426388 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng927\" (UniqueName: \"kubernetes.io/projected/99c48d2c-90bd-43ca-8b73-70851fca0c4b-kube-api-access-ng927\") pod \"maas-api-key-cleanup-29888715-9lfr9\" (UID: \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:00.426530 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.426500 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2\" (UID: \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:00.426689 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.426602 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74bv\" (UniqueName: \"kubernetes.io/projected/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-kube-api-access-h74bv\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7\" (UID: \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:00.426755 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.426733 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c48d2c-90bd-43ca-8b73-70851fca0c4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-9lfr9\" (UID: \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:00.426814 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.426785 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7\" (UID: \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:00.528843 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.528749 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2\" (UID: \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:00.529015 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.528848 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74bv\" (UniqueName: \"kubernetes.io/projected/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-kube-api-access-h74bv\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7\" (UID: \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:00.529015 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.528944 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c48d2c-90bd-43ca-8b73-70851fca0c4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-9lfr9\" (UID: \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:00.529015 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.528987 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7\" (UID: \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:00.529336 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.529315 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsfl\" (UniqueName: \"kubernetes.io/projected/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-kube-api-access-jnsfl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2\" (UID: \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:00.529434 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.529379 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng927\" (UniqueName: \"kubernetes.io/projected/99c48d2c-90bd-43ca-8b73-70851fca0c4b-kube-api-access-ng927\") pod \"maas-api-key-cleanup-29888715-9lfr9\" (UID: \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:00.529582 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.529548 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2\" (UID: \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:00.529720 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.529698 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7\" (UID: \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:00.529812 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.529789 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c48d2c-90bd-43ca-8b73-70851fca0c4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-9lfr9\" (UID: \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:00.554180 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.554121 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74bv\" (UniqueName: \"kubernetes.io/projected/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-kube-api-access-h74bv\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7\" (UID: \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:00.554540 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.554522 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng927\" (UniqueName: \"kubernetes.io/projected/99c48d2c-90bd-43ca-8b73-70851fca0c4b-kube-api-access-ng927\") pod \"maas-api-key-cleanup-29888715-9lfr9\" (UID: \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:00.554754 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.554731 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsfl\" (UniqueName: \"kubernetes.io/projected/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-kube-api-access-jnsfl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2\" (UID: \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:00.602691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.602651 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw"] Oct 30 01:15:00.615994 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.615962 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:00.619256 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.619230 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:00.622480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:00.622457 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:01.231507 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.224922 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl"] Oct 30 01:15:01.250126 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.249553 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl"] Oct 30 01:15:01.250126 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.249704 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.265059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.265031 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-481249-serving-cert\"" Oct 30 01:15:01.325732 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.325694 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9941506-88cf-4d6e-8736-719ef1438fe6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.325948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.325919 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-metrics-certs\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.326019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.325983 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmzhm\" (UniqueName: \"kubernetes.io/projected/d9941506-88cf-4d6e-8736-719ef1438fe6-kube-api-access-jmzhm\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.326095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.326079 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.373182 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.372799 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2"] Oct 30 01:15:01.376497 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:15:01.376467 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6cc7a3_70b9_439a_918a_aa92fd89a5b3.slice/crio-a1dce0b1c13603fa50730e9aa7bff6b81d4576f2353c5da578cb9cefe158c9db WatchSource:0}: Error finding container a1dce0b1c13603fa50730e9aa7bff6b81d4576f2353c5da578cb9cefe158c9db: Status 404 returned error can't find the container with id a1dce0b1c13603fa50730e9aa7bff6b81d4576f2353c5da578cb9cefe158c9db Oct 30 01:15:01.418233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.418158 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9"] Oct 30 01:15:01.420217 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:15:01.420164 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c48d2c_90bd_43ca_8b73_70851fca0c4b.slice/crio-ae1100a2d8ba8e1f8e3b14ebff32a679dd3dcb186df5b9feec4c456d8b5f82da WatchSource:0}: Error finding container ae1100a2d8ba8e1f8e3b14ebff32a679dd3dcb186df5b9feec4c456d8b5f82da: Status 404 returned error can't find the container with id ae1100a2d8ba8e1f8e3b14ebff32a679dd3dcb186df5b9feec4c456d8b5f82da Oct 30 01:15:01.428537 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.428506 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.428948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.428918 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9941506-88cf-4d6e-8736-719ef1438fe6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.429119 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.429099 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-metrics-certs\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.429197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.429155 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmzhm\" (UniqueName: \"kubernetes.io/projected/d9941506-88cf-4d6e-8736-719ef1438fe6-kube-api-access-jmzhm\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.430008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.429982 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9941506-88cf-4d6e-8736-719ef1438fe6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.432742 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.432683 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-metrics-certs\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.433351 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.433327 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.437267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.437220 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 01:15:01.451988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.451963 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmzhm\" (UniqueName: \"kubernetes.io/projected/d9941506-88cf-4d6e-8736-719ef1438fe6-kube-api-access-jmzhm\") pod \"maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.529969 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.529930 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-token\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-token\") pod \"b6441bb1-1412-4588-8f9a-200f1d0fee84\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " Oct 30 01:15:01.530179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.530052 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6441bb1-1412-4588-8f9a-200f1d0fee84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6441bb1-1412-4588-8f9a-200f1d0fee84-istiod-ca-cert\") pod \"b6441bb1-1412-4588-8f9a-200f1d0fee84\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " Oct 30 01:15:01.530179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.530133 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-podinfo\") pod \"b6441bb1-1412-4588-8f9a-200f1d0fee84\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " Oct 30 01:15:01.530314 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.530194 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-data\") pod \"b6441bb1-1412-4588-8f9a-200f1d0fee84\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " Oct 30 01:15:01.530373 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.530332 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-kube-api-access-nqrmr\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-kube-api-access-nqrmr\") pod \"b6441bb1-1412-4588-8f9a-200f1d0fee84\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " Oct 30 01:15:01.530436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.530382 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-socket\") pod \"b6441bb1-1412-4588-8f9a-200f1d0fee84\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " Oct 30 01:15:01.530544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.530520 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-credential-socket\") pod \"b6441bb1-1412-4588-8f9a-200f1d0fee84\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " Oct 30 01:15:01.530707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.530625 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-certs\") pod \"b6441bb1-1412-4588-8f9a-200f1d0fee84\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " Oct 30 01:15:01.530779 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.530762 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-envoy\") pod \"b6441bb1-1412-4588-8f9a-200f1d0fee84\" (UID: \"b6441bb1-1412-4588-8f9a-200f1d0fee84\") " Oct 30 01:15:01.531372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.531330 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-socket" pod "b6441bb1-1412-4588-8f9a-200f1d0fee84" (UID: "b6441bb1-1412-4588-8f9a-200f1d0fee84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:01.531793 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.531756 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:01.532278 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.532256 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-data" pod "b6441bb1-1412-4588-8f9a-200f1d0fee84" (UID: "b6441bb1-1412-4588-8f9a-200f1d0fee84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:01.533371 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.533349 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6441bb1-1412-4588-8f9a-200f1d0fee84-istiod-ca-cert" pod "b6441bb1-1412-4588-8f9a-200f1d0fee84" (UID: "b6441bb1-1412-4588-8f9a-200f1d0fee84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:01.533751 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.533713 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-token" pod "b6441bb1-1412-4588-8f9a-200f1d0fee84" (UID: "b6441bb1-1412-4588-8f9a-200f1d0fee84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:01.534135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.534094 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-certs" pod "b6441bb1-1412-4588-8f9a-200f1d0fee84" (UID: "b6441bb1-1412-4588-8f9a-200f1d0fee84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:01.534338 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.534296 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-credential-socket" pod "b6441bb1-1412-4588-8f9a-200f1d0fee84" (UID: "b6441bb1-1412-4588-8f9a-200f1d0fee84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:01.534502 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.534472 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-podinfo" pod "b6441bb1-1412-4588-8f9a-200f1d0fee84" (UID: "b6441bb1-1412-4588-8f9a-200f1d0fee84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:15:01.536224 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.536200 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-envoy" pod "b6441bb1-1412-4588-8f9a-200f1d0fee84" (UID: "b6441bb1-1412-4588-8f9a-200f1d0fee84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:01.536384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.536353 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-kube-api-access-nqrmr" pod "b6441bb1-1412-4588-8f9a-200f1d0fee84" (UID: "b6441bb1-1412-4588-8f9a-200f1d0fee84"). InnerVolumeSpecName "kube-api-access-nqrmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:01.536993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.536967 4927 generic.go:356] "Generic (PLEG): container finished" podID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerID="736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029" exitCode=0 Oct 30 01:15:01.537091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.537024 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" Oct 30 01:15:01.537091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.537052 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" event={"ID":"b6441bb1-1412-4588-8f9a-200f1d0fee84","Type":"ContainerDied","Data":"736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029"} Oct 30 01:15:01.537187 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.537115 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92" event={"ID":"b6441bb1-1412-4588-8f9a-200f1d0fee84","Type":"ContainerDied","Data":"cb7f945659f61969be08264f4140f39c5229116853b9bbe2ce4baed79dea3c2a"} Oct 30 01:15:01.537187 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.537146 4927 scope.go:122] "RemoveContainer" containerID="736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029" Oct 30 01:15:01.538918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.538894 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" event={"ID":"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3","Type":"ContainerStarted","Data":"a1dce0b1c13603fa50730e9aa7bff6b81d4576f2353c5da578cb9cefe158c9db"} Oct 30 01:15:01.540646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.540621 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" event={"ID":"99c48d2c-90bd-43ca-8b73-70851fca0c4b","Type":"ContainerStarted","Data":"ae1100a2d8ba8e1f8e3b14ebff32a679dd3dcb186df5b9feec4c456d8b5f82da"} Oct 30 01:15:01.551255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.551235 4927 scope.go:122] "RemoveContainer" containerID="736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029" Oct 30 01:15:01.551659 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:01.551633 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029\": container with ID starting with 736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029 not found: ID does not exist" containerID="736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029" Oct 30 01:15:01.551760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.551670 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029"} err="failed to get container status \"736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029\": rpc error: code = NotFound desc = could not find container \"736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029\": container with ID starting with 736fcdd539d98d3590760e3a09b9830479d5c5ed876bdfd2445be50bbc6dc029 not found: ID does not exist" Oct 30 01:15:01.574940 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.574915 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:01.584122 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.584096 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7"] Oct 30 01:15:01.584842 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:15:01.584817 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0623ac0b_7aff_435d_9bb9_7031e2ebfcc4.slice/crio-d094950072852d8eb96f39dee1037a94c762fb93effdccf38b76ebbfccd1e32e WatchSource:0}: Error finding container d094950072852d8eb96f39dee1037a94c762fb93effdccf38b76ebbfccd1e32e: Status 404 returned error can't find the container with id d094950072852d8eb96f39dee1037a94c762fb93effdccf38b76ebbfccd1e32e Oct 30 01:15:01.599729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.599557 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92"] Oct 30 01:15:01.606898 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.606870 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mkm92"] Oct 30 01:15:01.633951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.633913 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqrmr\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-kube-api-access-nqrmr\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:01.633951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.633951 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:01.633951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.633968 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:01.634220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.633983 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:01.634220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.633996 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:01.634220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.634008 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6441bb1-1412-4588-8f9a-200f1d0fee84-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:01.634220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.634020 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:01.634220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:01.634034 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6441bb1-1412-4588-8f9a-200f1d0fee84-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:02.134075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.134023 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm"] Oct 30 01:15:02.177981 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.177945 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" Oct 30 01:15:02.211157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.210543 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" Oct 30 01:15:02.212499 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.212283 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" containerName="istio-proxy" Oct 30 01:15:02.228915 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.228881 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.246948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.246554 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d1ae36-openshift-default-dockercfg-54tgv\"" Oct 30 01:15:02.250837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.250612 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm"] Oct 30 01:15:02.257363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.257191 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl"] Oct 30 01:15:02.363600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.354879 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-credential-socket\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.364058 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.364011 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-envoy\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.370599 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.364338 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-socket\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.370882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.370848 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345e5ad3-c558-478f-a4a9-641dec5af039-istiod-ca-cert\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.371111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.371089 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-data\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.386046 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.385960 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345e5ad3-c558-478f-a4a9-641dec5af039-istio-podinfo\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.386512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.386486 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxrtk\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-kube-api-access-qxrtk\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.386731 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.386704 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-certs\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.394149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.394121 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-istio-token\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.497111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.497070 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345e5ad3-c558-478f-a4a9-641dec5af039-istio-podinfo\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.514312 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.511018 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345e5ad3-c558-478f-a4a9-641dec5af039-istio-podinfo\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.514849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.514824 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxrtk\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-kube-api-access-qxrtk\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.517382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.517070 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-certs\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.517382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.517135 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-istio-token\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.517382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.517314 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-credential-socket\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.517659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.517443 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-envoy\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.517659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.517492 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-socket\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.517659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.517536 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345e5ad3-c558-478f-a4a9-641dec5af039-istiod-ca-cert\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.517819 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.517668 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-data\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.518401 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.518302 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-data\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.518602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.518563 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-socket\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.519091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.519034 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-certs\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.519425 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.519322 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345e5ad3-c558-478f-a4a9-641dec5af039-istiod-ca-cert\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.519848 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.519815 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-credential-socket\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.522326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.522287 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-envoy\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.542228 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.542184 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxrtk\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-kube-api-access-qxrtk\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.543240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.543213 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-istio-token\") pod \"e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.553498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.553461 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" event={"ID":"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4","Type":"ContainerStarted","Data":"d094950072852d8eb96f39dee1037a94c762fb93effdccf38b76ebbfccd1e32e"} Oct 30 01:15:02.555820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.555792 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" event={"ID":"d9941506-88cf-4d6e-8736-719ef1438fe6","Type":"ContainerStarted","Data":"ea8d967668c91e1b2305a5c5cee2259465fce9ccb68b66cb3d3392e2c06e86c3"} Oct 30 01:15:02.581880 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.581839 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:02.937803 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:15:02.937759 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345e5ad3_c558_478f_a4a9_641dec5af039.slice/crio-6f990dd91695d4095bafd8658f78dd1f77b39577237c4788b4bc8471ed5ccb79 WatchSource:0}: Error finding container 6f990dd91695d4095bafd8658f78dd1f77b39577237c4788b4bc8471ed5ccb79: Status 404 returned error can't find the container with id 6f990dd91695d4095bafd8658f78dd1f77b39577237c4788b4bc8471ed5ccb79 Oct 30 01:15:02.939430 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.938720 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm"] Oct 30 01:15:02.943131 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.943082 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:02.943267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.943184 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:02.943267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:02.943228 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:03.591962 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:03.591925 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6441bb1-1412-4588-8f9a-200f1d0fee84" path="/var/lib/kubelet/pods/b6441bb1-1412-4588-8f9a-200f1d0fee84/volumes" Oct 30 01:15:03.593504 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:03.593468 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" event={"ID":"345e5ad3-c558-478f-a4a9-641dec5af039","Type":"ContainerStarted","Data":"a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e"} Oct 30 01:15:03.593772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:03.593744 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" event={"ID":"345e5ad3-c558-478f-a4a9-641dec5af039","Type":"ContainerStarted","Data":"6f990dd91695d4095bafd8658f78dd1f77b39577237c4788b4bc8471ed5ccb79"} Oct 30 01:15:03.596794 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:03.596769 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" event={"ID":"d9941506-88cf-4d6e-8736-719ef1438fe6","Type":"ContainerStarted","Data":"52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4"} Oct 30 01:15:03.597014 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:03.596999 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:03.661114 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:03.661044 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" podStartSLOduration=2.661023078 podStartE2EDuration="2.661023078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.648882051 +0000 UTC m=+2442.220808274" watchObservedRunningTime="2026-10-30 01:15:03.661023078 +0000 UTC m=+2442.232949302" Oct 30 01:15:03.696855 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:03.696794 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" podStartSLOduration=3.205477594 podStartE2EDuration="3.69677337s" totalImagesPullingTime="491.295776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.689060346 +0000 UTC m=+2442.260986569" watchObservedRunningTime="2026-10-30 01:15:03.69677337 +0000 UTC m=+2442.268699593" Oct 30 01:15:04.453274 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:04.453215 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:15:04.583373 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:04.583274 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:04.590678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:04.590646 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:04.611236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:04.611205 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:04.612367 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:04.612343 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:15:06.630677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:06.630633 4927 generic.go:356] "Generic (PLEG): container finished" podID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerID="b3f198a1cc324f192fe7cd064b39b6936c41c0e86b15f80dbee3a1b6951cf059" exitCode=60 Oct 30 01:15:06.631153 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:06.630714 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" event={"ID":"99c48d2c-90bd-43ca-8b73-70851fca0c4b","Type":"ContainerDied","Data":"b3f198a1cc324f192fe7cd064b39b6936c41c0e86b15f80dbee3a1b6951cf059"} Oct 30 01:15:06.631153 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:06.631101 4927 scope.go:122] "RemoveContainer" containerID="b3f198a1cc324f192fe7cd064b39b6936c41c0e86b15f80dbee3a1b6951cf059" Oct 30 01:15:06.633046 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:06.633012 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" event={"ID":"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3","Type":"ContainerStarted","Data":"4df2a68b978fc45e0cb6fa3d70c97295d330b45c3aafe3970ad5d6cf3067aea0"} Oct 30 01:15:06.634828 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:06.634799 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" event={"ID":"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4","Type":"ContainerStarted","Data":"c7c24f0ab91918c6960b9c3ea41529bcd6efb52371dcb46812b434f32cf681d0"} Oct 30 01:15:06.717831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:06.717775 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" podStartSLOduration=2.634549082 podStartE2EDuration="6.717759596s" totalImagesPullingTime="4.083210514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.716208132 +0000 UTC m=+2445.288134355" watchObservedRunningTime="2026-10-30 01:15:06.717759596 +0000 UTC m=+2445.289685850" Oct 30 01:15:06.760531 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:06.760466 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" podStartSLOduration=2.837278109 podStartE2EDuration="6.76045162s" totalImagesPullingTime="3.923173511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.755306656 +0000 UTC m=+2445.327232875" watchObservedRunningTime="2026-10-30 01:15:06.76045162 +0000 UTC m=+2445.332377876" Oct 30 01:15:07.645624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.645555 4927 generic.go:356] "Generic (PLEG): container finished" podID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerID="c7c24f0ab91918c6960b9c3ea41529bcd6efb52371dcb46812b434f32cf681d0" exitCode=60 Oct 30 01:15:07.645624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.645601 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" event={"ID":"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4","Type":"ContainerDied","Data":"c7c24f0ab91918c6960b9c3ea41529bcd6efb52371dcb46812b434f32cf681d0"} Oct 30 01:15:07.646132 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.646044 4927 scope.go:122] "RemoveContainer" containerID="c7c24f0ab91918c6960b9c3ea41529bcd6efb52371dcb46812b434f32cf681d0" Oct 30 01:15:07.647991 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.647964 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" event={"ID":"99c48d2c-90bd-43ca-8b73-70851fca0c4b","Type":"ContainerStarted","Data":"39014976d9a4fab0f39bab351e17f574d81ebd66d0934ef90c8c78e2cd51d621"} Oct 30 01:15:07.649920 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.649893 4927 generic.go:356] "Generic (PLEG): container finished" podID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerID="4df2a68b978fc45e0cb6fa3d70c97295d330b45c3aafe3970ad5d6cf3067aea0" exitCode=60 Oct 30 01:15:07.650032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.649932 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" event={"ID":"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3","Type":"ContainerDied","Data":"4df2a68b978fc45e0cb6fa3d70c97295d330b45c3aafe3970ad5d6cf3067aea0"} Oct 30 01:15:07.650310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.650294 4927 scope.go:122] "RemoveContainer" containerID="4df2a68b978fc45e0cb6fa3d70c97295d330b45c3aafe3970ad5d6cf3067aea0" Oct 30 01:15:07.732236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.732186 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" podStartSLOduration=3.687385076 podStartE2EDuration="7.732171205s" totalImagesPullingTime="4.044786129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:07.727414584 +0000 UTC m=+2446.299340817" watchObservedRunningTime="2026-10-30 01:15:07.732171205 +0000 UTC m=+2446.304097427" Oct 30 01:15:07.855490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.855455 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w"] Oct 30 01:15:07.855818 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:07.855719 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" containerID="cri-o://4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc" gracePeriod=30 Oct 30 01:15:08.037586 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.037478 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf"] Oct 30 01:15:08.042820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.042792 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:08.060284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.060252 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d1ae36\"" Oct 30 01:15:08.065402 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.065364 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf"] Oct 30 01:15:08.075005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.074966 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l"] Oct 30 01:15:08.075692 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.075659 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" containerID="cri-o://f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778" gracePeriod=30 Oct 30 01:15:08.079660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.079631 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc2c4\" (UniqueName: \"kubernetes.io/projected/8a137c2e-bce6-4430-9b40-751f28ed58c4-kube-api-access-wc2c4\") pod \"payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf\" (UID: \"8a137c2e-bce6-4430-9b40-751f28ed58c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:08.080770 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.080741 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a137c2e-bce6-4430-9b40-751f28ed58c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf\" (UID: \"8a137c2e-bce6-4430-9b40-751f28ed58c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:08.089842 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.089816 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr"] Oct 30 01:15:08.090422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.090387 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" containerID="cri-o://44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a" gracePeriod=30 Oct 30 01:15:08.182907 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.182872 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc2c4\" (UniqueName: \"kubernetes.io/projected/8a137c2e-bce6-4430-9b40-751f28ed58c4-kube-api-access-wc2c4\") pod \"payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf\" (UID: \"8a137c2e-bce6-4430-9b40-751f28ed58c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:08.183221 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.183197 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a137c2e-bce6-4430-9b40-751f28ed58c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf\" (UID: \"8a137c2e-bce6-4430-9b40-751f28ed58c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:08.183993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.183966 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a137c2e-bce6-4430-9b40-751f28ed58c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf\" (UID: \"8a137c2e-bce6-4430-9b40-751f28ed58c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:08.206930 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.206890 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc2c4\" (UniqueName: \"kubernetes.io/projected/8a137c2e-bce6-4430-9b40-751f28ed58c4-kube-api-access-wc2c4\") pod \"payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf\" (UID: \"8a137c2e-bce6-4430-9b40-751f28ed58c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:08.360919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.360880 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:08.673435 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.671250 4927 generic.go:356] "Generic (PLEG): container finished" podID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerID="39014976d9a4fab0f39bab351e17f574d81ebd66d0934ef90c8c78e2cd51d621" exitCode=60 Oct 30 01:15:08.673435 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.671556 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" event={"ID":"99c48d2c-90bd-43ca-8b73-70851fca0c4b","Type":"ContainerDied","Data":"39014976d9a4fab0f39bab351e17f574d81ebd66d0934ef90c8c78e2cd51d621"} Oct 30 01:15:08.673435 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.671640 4927 scope.go:122] "RemoveContainer" containerID="b3f198a1cc324f192fe7cd064b39b6936c41c0e86b15f80dbee3a1b6951cf059" Oct 30 01:15:08.673435 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.672198 4927 scope.go:122] "RemoveContainer" containerID="39014976d9a4fab0f39bab351e17f574d81ebd66d0934ef90c8c78e2cd51d621" Oct 30 01:15:08.673435 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:08.672655 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-9lfr9_odh-ai-gateway-infra(99c48d2c-90bd-43ca-8b73-70851fca0c4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" Oct 30 01:15:08.678556 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.678523 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" event={"ID":"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3","Type":"ContainerStarted","Data":"790597c969f57ea35393808dbe0355b95daeecc10d0a0c7c11ede3a00a8c757c"} Oct 30 01:15:08.683650 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.683606 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" event={"ID":"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4","Type":"ContainerStarted","Data":"c30b86785b6dd8145facdf4a0b75a30485cc22b710f95e18290cd6f6b33fb6f0"} Oct 30 01:15:08.953778 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:08.953748 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf"] Oct 30 01:15:08.953909 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:15:08.953746 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a137c2e_bce6_4430_9b40_751f28ed58c4.slice/crio-3c79ebf8486f357bedff8c98c31cba3c412bd2458f57a21cadc9cef8d994a987 WatchSource:0}: Error finding container 3c79ebf8486f357bedff8c98c31cba3c412bd2458f57a21cadc9cef8d994a987: Status 404 returned error can't find the container with id 3c79ebf8486f357bedff8c98c31cba3c412bd2458f57a21cadc9cef8d994a987 Oct 30 01:15:09.617059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.616965 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:15:09.700264 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.700214 4927 generic.go:356] "Generic (PLEG): container finished" podID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerID="790597c969f57ea35393808dbe0355b95daeecc10d0a0c7c11ede3a00a8c757c" exitCode=60 Oct 30 01:15:09.700797 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.700768 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" event={"ID":"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3","Type":"ContainerDied","Data":"790597c969f57ea35393808dbe0355b95daeecc10d0a0c7c11ede3a00a8c757c"} Oct 30 01:15:09.700939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.700879 4927 scope.go:122] "RemoveContainer" containerID="4df2a68b978fc45e0cb6fa3d70c97295d330b45c3aafe3970ad5d6cf3067aea0" Oct 30 01:15:09.701305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.701256 4927 scope.go:122] "RemoveContainer" containerID="790597c969f57ea35393808dbe0355b95daeecc10d0a0c7c11ede3a00a8c757c" Oct 30 01:15:09.702041 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:09.702004 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2_odh-ai-gateway-infra(fd6cc7a3-70b9-439a-918a-aa92fd89a5b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" Oct 30 01:15:09.704062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.704034 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" event={"ID":"8a137c2e-bce6-4430-9b40-751f28ed58c4","Type":"ContainerStarted","Data":"324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2"} Oct 30 01:15:09.704266 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.704246 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" event={"ID":"8a137c2e-bce6-4430-9b40-751f28ed58c4","Type":"ContainerStarted","Data":"3c79ebf8486f357bedff8c98c31cba3c412bd2458f57a21cadc9cef8d994a987"} Oct 30 01:15:09.704410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.704387 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:09.709615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.706603 4927 generic.go:356] "Generic (PLEG): container finished" podID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerID="c30b86785b6dd8145facdf4a0b75a30485cc22b710f95e18290cd6f6b33fb6f0" exitCode=60 Oct 30 01:15:09.709615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.706710 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" event={"ID":"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4","Type":"ContainerDied","Data":"c30b86785b6dd8145facdf4a0b75a30485cc22b710f95e18290cd6f6b33fb6f0"} Oct 30 01:15:09.709615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.707080 4927 scope.go:122] "RemoveContainer" containerID="c30b86785b6dd8145facdf4a0b75a30485cc22b710f95e18290cd6f6b33fb6f0" Oct 30 01:15:09.709615 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:09.707502 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7_odh-ai-gateway-infra(0623ac0b-7aff-435d-9bb9-7031e2ebfcc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" Oct 30 01:15:09.709615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.709085 4927 scope.go:122] "RemoveContainer" containerID="39014976d9a4fab0f39bab351e17f574d81ebd66d0934ef90c8c78e2cd51d621" Oct 30 01:15:09.709615 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:09.709360 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-9lfr9_odh-ai-gateway-infra(99c48d2c-90bd-43ca-8b73-70851fca0c4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" Oct 30 01:15:09.728744 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.728715 4927 scope.go:122] "RemoveContainer" containerID="c7c24f0ab91918c6960b9c3ea41529bcd6efb52371dcb46812b434f32cf681d0" Oct 30 01:15:09.823859 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:09.823805 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" podStartSLOduration=1.823788799 podStartE2EDuration="1.823788799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:09.818520581 +0000 UTC m=+2448.390446805" watchObservedRunningTime="2026-10-30 01:15:09.823788799 +0000 UTC m=+2448.395715021" Oct 30 01:15:10.483763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:10.483724 4927 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:15:10.483987 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:10.483794 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:15:10.577179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:10.577139 4927 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:15:10.577349 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:10.577205 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:15:10.806652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:10.805982 4927 scope.go:122] "RemoveContainer" containerID="790597c969f57ea35393808dbe0355b95daeecc10d0a0c7c11ede3a00a8c757c" Oct 30 01:15:10.806652 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:10.806460 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2_odh-ai-gateway-infra(fd6cc7a3-70b9-439a-918a-aa92fd89a5b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" Oct 30 01:15:10.820599 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:10.815783 4927 scope.go:122] "RemoveContainer" containerID="c30b86785b6dd8145facdf4a0b75a30485cc22b710f95e18290cd6f6b33fb6f0" Oct 30 01:15:10.820599 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:10.816212 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7_odh-ai-gateway-infra(0623ac0b-7aff-435d-9bb9-7031e2ebfcc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" Oct 30 01:15:13.289732 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.289087 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 01:15:13.419292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.419183 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-kube-api-access-5rmzd\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-kube-api-access-5rmzd\") pod \"4316fd2e-adab-4e5e-9b8d-c139100038a4\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " Oct 30 01:15:13.419292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.419262 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-podinfo\") pod \"4316fd2e-adab-4e5e-9b8d-c139100038a4\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " Oct 30 01:15:13.419543 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.419349 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-certs\") pod \"4316fd2e-adab-4e5e-9b8d-c139100038a4\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " Oct 30 01:15:13.419543 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.419389 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-credential-socket\") pod \"4316fd2e-adab-4e5e-9b8d-c139100038a4\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " Oct 30 01:15:13.419543 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.419522 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-token\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-token\") pod \"4316fd2e-adab-4e5e-9b8d-c139100038a4\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " Oct 30 01:15:13.419796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.419600 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4316fd2e-adab-4e5e-9b8d-c139100038a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4316fd2e-adab-4e5e-9b8d-c139100038a4-istiod-ca-cert\") pod \"4316fd2e-adab-4e5e-9b8d-c139100038a4\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " Oct 30 01:15:13.419796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.419739 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-socket\") pod \"4316fd2e-adab-4e5e-9b8d-c139100038a4\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " Oct 30 01:15:13.419909 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.419801 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-data\") pod \"4316fd2e-adab-4e5e-9b8d-c139100038a4\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " Oct 30 01:15:13.419909 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.419840 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-envoy\") pod \"4316fd2e-adab-4e5e-9b8d-c139100038a4\" (UID: \"4316fd2e-adab-4e5e-9b8d-c139100038a4\") " Oct 30 01:15:13.420334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.420301 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-credential-socket" pod "4316fd2e-adab-4e5e-9b8d-c139100038a4" (UID: "4316fd2e-adab-4e5e-9b8d-c139100038a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.421076 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.421046 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-certs" pod "4316fd2e-adab-4e5e-9b8d-c139100038a4" (UID: "4316fd2e-adab-4e5e-9b8d-c139100038a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.421201 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.421054 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-data" pod "4316fd2e-adab-4e5e-9b8d-c139100038a4" (UID: "4316fd2e-adab-4e5e-9b8d-c139100038a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.421335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.421115 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4316fd2e-adab-4e5e-9b8d-c139100038a4-istiod-ca-cert" pod "4316fd2e-adab-4e5e-9b8d-c139100038a4" (UID: "4316fd2e-adab-4e5e-9b8d-c139100038a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:13.421946 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.421919 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.422070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.422052 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.422183 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.422169 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4316fd2e-adab-4e5e-9b8d-c139100038a4-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.422266 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.422255 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.427942 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.427912 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-envoy" pod "4316fd2e-adab-4e5e-9b8d-c139100038a4" (UID: "4316fd2e-adab-4e5e-9b8d-c139100038a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.430109 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.430081 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-kube-api-access-5rmzd" pod "4316fd2e-adab-4e5e-9b8d-c139100038a4" (UID: "4316fd2e-adab-4e5e-9b8d-c139100038a4"). InnerVolumeSpecName "kube-api-access-5rmzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:13.440677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.436174 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-socket" pod "4316fd2e-adab-4e5e-9b8d-c139100038a4" (UID: "4316fd2e-adab-4e5e-9b8d-c139100038a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.440677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.437794 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-token" pod "4316fd2e-adab-4e5e-9b8d-c139100038a4" (UID: "4316fd2e-adab-4e5e-9b8d-c139100038a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:13.441723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.441676 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-podinfo" pod "4316fd2e-adab-4e5e-9b8d-c139100038a4" (UID: "4316fd2e-adab-4e5e-9b8d-c139100038a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:15:13.458321 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.458287 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 01:15:13.499091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.499062 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 01:15:13.523415 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.523379 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-credential-socket\") pod \"3762592f-23e3-476a-a79d-29837a453845\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " Oct 30 01:15:13.523707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.523484 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-envoy\") pod \"3762592f-23e3-476a-a79d-29837a453845\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " Oct 30 01:15:13.523707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.523538 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-istio-token\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-istio-token\") pod \"3762592f-23e3-476a-a79d-29837a453845\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " Oct 30 01:15:13.523707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.523641 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-kube-api-access-jb99c\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-kube-api-access-jb99c\") pod \"3762592f-23e3-476a-a79d-29837a453845\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " Oct 30 01:15:13.523707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.523688 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3762592f-23e3-476a-a79d-29837a453845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3762592f-23e3-476a-a79d-29837a453845-istiod-ca-cert\") pod \"3762592f-23e3-476a-a79d-29837a453845\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " Oct 30 01:15:13.524082 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.523741 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-credential-socket" pod "3762592f-23e3-476a-a79d-29837a453845" (UID: "3762592f-23e3-476a-a79d-29837a453845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.524082 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.523768 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-data\") pod \"3762592f-23e3-476a-a79d-29837a453845\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " Oct 30 01:15:13.524082 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.523878 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-socket\") pod \"3762592f-23e3-476a-a79d-29837a453845\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " Oct 30 01:15:13.524082 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.523982 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3762592f-23e3-476a-a79d-29837a453845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3762592f-23e3-476a-a79d-29837a453845-istio-podinfo\") pod \"3762592f-23e3-476a-a79d-29837a453845\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " Oct 30 01:15:13.524283 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.524248 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-certs\") pod \"3762592f-23e3-476a-a79d-29837a453845\" (UID: \"3762592f-23e3-476a-a79d-29837a453845\") " Oct 30 01:15:13.524422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.524377 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-data" pod "3762592f-23e3-476a-a79d-29837a453845" (UID: "3762592f-23e3-476a-a79d-29837a453845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.524692 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.524669 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3762592f-23e3-476a-a79d-29837a453845-istiod-ca-cert" pod "3762592f-23e3-476a-a79d-29837a453845" (UID: "3762592f-23e3-476a-a79d-29837a453845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:13.524877 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.524857 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-certs" pod "3762592f-23e3-476a-a79d-29837a453845" (UID: "3762592f-23e3-476a-a79d-29837a453845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.525040 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525023 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-socket" pod "3762592f-23e3-476a-a79d-29837a453845" (UID: "3762592f-23e3-476a-a79d-29837a453845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.525265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525241 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.525416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525380 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rmzd\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-kube-api-access-5rmzd\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.525416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525399 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.525416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525414 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3762592f-23e3-476a-a79d-29837a453845-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.525696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525427 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.525696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525438 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.525696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525446 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.525696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525457 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.525696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525466 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4316fd2e-adab-4e5e-9b8d-c139100038a4-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.525696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.525475 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.527007 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.526978 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-kube-api-access-jb99c" pod "3762592f-23e3-476a-a79d-29837a453845" (UID: "3762592f-23e3-476a-a79d-29837a453845"). InnerVolumeSpecName "kube-api-access-jb99c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:13.527878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.527845 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-envoy" pod "3762592f-23e3-476a-a79d-29837a453845" (UID: "3762592f-23e3-476a-a79d-29837a453845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.529083 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.529061 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3762592f-23e3-476a-a79d-29837a453845-istio-podinfo" pod "3762592f-23e3-476a-a79d-29837a453845" (UID: "3762592f-23e3-476a-a79d-29837a453845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:15:13.530006 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.529986 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-istio-token" pod "3762592f-23e3-476a-a79d-29837a453845" (UID: "3762592f-23e3-476a-a79d-29837a453845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:13.626791 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.626741 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-credential-socket\") pod \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " Oct 30 01:15:13.626979 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.626873 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-token\") pod \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " Oct 30 01:15:13.626979 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.626957 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-kube-api-access-svtvl\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-kube-api-access-svtvl\") pod \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " Oct 30 01:15:13.627109 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.626994 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-certs\") pod \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " Oct 30 01:15:13.627109 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.627073 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-credential-socket" pod "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" (UID: "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.627228 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.627083 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-podinfo\") pod \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " Oct 30 01:15:13.627228 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.627160 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istiod-ca-cert\") pod \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " Oct 30 01:15:13.627342 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.627236 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-socket\") pod \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " Oct 30 01:15:13.627395 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.627343 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-certs" pod "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" (UID: "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.627795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.627671 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istiod-ca-cert" pod "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" (UID: "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:13.627795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.627747 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-socket" pod "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" (UID: "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.627989 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.627945 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-envoy\") pod \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " Oct 30 01:15:13.628056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.627986 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-data\") pod \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\" (UID: \"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4\") " Oct 30 01:15:13.628434 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.628393 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-data" pod "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" (UID: "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.629461 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629436 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.629565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629466 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.629651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629583 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3762592f-23e3-476a-a79d-29837a453845-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.629651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629629 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.629651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629642 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.629821 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629655 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3762592f-23e3-476a-a79d-29837a453845-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.629821 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629669 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.629821 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629680 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.629821 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629693 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb99c\" (UniqueName: \"kubernetes.io/projected/3762592f-23e3-476a-a79d-29837a453845-kube-api-access-jb99c\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.630025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629842 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-podinfo" pod "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" (UID: "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:15:13.630025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.629994 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-token" pod "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" (UID: "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:13.631127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.631089 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-kube-api-access-svtvl" pod "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" (UID: "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4"). InnerVolumeSpecName "kube-api-access-svtvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:13.631507 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.631485 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-envoy" pod "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" (UID: "8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:13.731459 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.731367 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.731459 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.731404 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.731459 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.731419 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtvl\" (UniqueName: \"kubernetes.io/projected/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-kube-api-access-svtvl\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.731459 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.731434 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:13.845802 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.845767 4927 generic.go:356] "Generic (PLEG): container finished" podID="3762592f-23e3-476a-a79d-29837a453845" containerID="44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a" exitCode=0 Oct 30 01:15:13.846030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.845813 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" Oct 30 01:15:13.846030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.845844 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" event={"ID":"3762592f-23e3-476a-a79d-29837a453845","Type":"ContainerDied","Data":"44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a"} Oct 30 01:15:13.846030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.845905 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr" event={"ID":"3762592f-23e3-476a-a79d-29837a453845","Type":"ContainerDied","Data":"1f3227ec335b09f29cbd1e448a367e5860ac7d45f231bbc0edb3f69b2fa2692d"} Oct 30 01:15:13.846030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.845934 4927 scope.go:122] "RemoveContainer" containerID="44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a" Oct 30 01:15:13.847727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.847704 4927 generic.go:356] "Generic (PLEG): container finished" podID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerID="f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778" exitCode=0 Oct 30 01:15:13.847875 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.847858 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" Oct 30 01:15:13.848305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.848171 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" event={"ID":"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4","Type":"ContainerDied","Data":"f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778"} Oct 30 01:15:13.848305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.848220 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l" event={"ID":"8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4","Type":"ContainerDied","Data":"f790a1eb8972263ec4a5048d0b4250bf16e7a395bd1b203133f1434e49b81b81"} Oct 30 01:15:13.850016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.849993 4927 generic.go:356] "Generic (PLEG): container finished" podID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerID="4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc" exitCode=0 Oct 30 01:15:13.850132 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.850041 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" event={"ID":"4316fd2e-adab-4e5e-9b8d-c139100038a4","Type":"ContainerDied","Data":"4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc"} Oct 30 01:15:13.850132 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.850070 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" event={"ID":"4316fd2e-adab-4e5e-9b8d-c139100038a4","Type":"ContainerDied","Data":"aafd360d302980b1472965201f3a43c64ed7077d3b935c545584bf1557d703d4"} Oct 30 01:15:13.850132 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.850128 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w" Oct 30 01:15:13.858889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.858779 4927 scope.go:122] "RemoveContainer" containerID="44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a" Oct 30 01:15:13.859146 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:13.859124 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a\": container with ID starting with 44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a not found: ID does not exist" containerID="44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a" Oct 30 01:15:13.859243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.859156 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a"} err="failed to get container status \"44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a\": rpc error: code = NotFound desc = could not find container \"44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a\": container with ID starting with 44e0c6af0910517a7d349674196875076b734289d063744f903dc37c14e5ab7a not found: ID does not exist" Oct 30 01:15:13.859243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.859174 4927 scope.go:122] "RemoveContainer" containerID="f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778" Oct 30 01:15:13.870521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.870498 4927 scope.go:122] "RemoveContainer" containerID="f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778" Oct 30 01:15:13.871009 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:13.870984 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778\": container with ID starting with f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778 not found: ID does not exist" containerID="f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778" Oct 30 01:15:13.871148 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.871016 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778"} err="failed to get container status \"f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778\": rpc error: code = NotFound desc = could not find container \"f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778\": container with ID starting with f62a2327a75027a5c0c8e00dfdb8a86b3174f64e9f50d5d0ff21b403d28ec778 not found: ID does not exist" Oct 30 01:15:13.871148 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.871033 4927 scope.go:122] "RemoveContainer" containerID="4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc" Oct 30 01:15:13.903263 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.903224 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l"] Oct 30 01:15:13.914494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.914463 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-tlz7l"] Oct 30 01:15:13.917094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.917073 4927 scope.go:122] "RemoveContainer" containerID="4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc" Oct 30 01:15:13.917396 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:13.917374 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc\": container with ID starting with 4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc not found: ID does not exist" containerID="4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc" Oct 30 01:15:13.917478 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.917406 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc"} err="failed to get container status \"4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc\": rpc error: code = NotFound desc = could not find container \"4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc\": container with ID starting with 4ad7123cd300369cff91888798d8194b07336a7d4266d8f5994d3784162da5cc not found: ID does not exist" Oct 30 01:15:13.941648 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.941612 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr"] Oct 30 01:15:13.946835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.946808 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-rpfsr"] Oct 30 01:15:13.976995 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.976958 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w"] Oct 30 01:15:13.981996 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:13.981925 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-qpq5w"] Oct 30 01:15:15.574070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:15.574031 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3762592f-23e3-476a-a79d-29837a453845" path="/var/lib/kubelet/pods/3762592f-23e3-476a-a79d-29837a453845/volumes" Oct 30 01:15:15.574792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:15.574772 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" path="/var/lib/kubelet/pods/4316fd2e-adab-4e5e-9b8d-c139100038a4/volumes" Oct 30 01:15:15.575470 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:15.575451 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" path="/var/lib/kubelet/pods/8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4/volumes" Oct 30 01:15:15.633139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:15.633102 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" podUID="89a3994e-d292-4fe4-ba1f-e61e80731b21" containerName="main" containerID="cri-o://91fa780c518c4d86ac42de5149d2cda3fdf1d29398eb6f5bb054f384f9213ac4" gracePeriod=45 Oct 30 01:15:15.887954 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:15.874323 4927 generic.go:356] "Generic (PLEG): container finished" podID="89a3994e-d292-4fe4-ba1f-e61e80731b21" containerID="91fa780c518c4d86ac42de5149d2cda3fdf1d29398eb6f5bb054f384f9213ac4" exitCode=0 Oct 30 01:15:15.887954 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:15.874425 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" event={"ID":"89a3994e-d292-4fe4-ba1f-e61e80731b21","Type":"ContainerDied","Data":"91fa780c518c4d86ac42de5149d2cda3fdf1d29398eb6f5bb054f384f9213ac4"} Oct 30 01:15:15.956618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:15.956561 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:15:16.071326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.071295 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-dshm\") pod \"89a3994e-d292-4fe4-ba1f-e61e80731b21\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " Oct 30 01:15:16.071503 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.071486 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a3994e-d292-4fe4-ba1f-e61e80731b21-kube-api-access-pz9tt\" (UniqueName: \"kubernetes.io/projected/89a3994e-d292-4fe4-ba1f-e61e80731b21-kube-api-access-pz9tt\") pod \"89a3994e-d292-4fe4-ba1f-e61e80731b21\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " Oct 30 01:15:16.071628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.071528 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-tmp-dir\") pod \"89a3994e-d292-4fe4-ba1f-e61e80731b21\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " Oct 30 01:15:16.071758 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.071731 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-model-cache\") pod \"89a3994e-d292-4fe4-ba1f-e61e80731b21\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " Oct 30 01:15:16.071835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.071823 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a3994e-d292-4fe4-ba1f-e61e80731b21-tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3994e-d292-4fe4-ba1f-e61e80731b21-tls-certs\") pod \"89a3994e-d292-4fe4-ba1f-e61e80731b21\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " Oct 30 01:15:16.071886 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.071833 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-tmp-dir" pod "89a3994e-d292-4fe4-ba1f-e61e80731b21" (UID: "89a3994e-d292-4fe4-ba1f-e61e80731b21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:16.071944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.071895 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-home\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-home\") pod \"89a3994e-d292-4fe4-ba1f-e61e80731b21\" (UID: \"89a3994e-d292-4fe4-ba1f-e61e80731b21\") " Oct 30 01:15:16.072027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.072001 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-model-cache" pod "89a3994e-d292-4fe4-ba1f-e61e80731b21" (UID: "89a3994e-d292-4fe4-ba1f-e61e80731b21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:16.072197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.072168 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-home" pod "89a3994e-d292-4fe4-ba1f-e61e80731b21" (UID: "89a3994e-d292-4fe4-ba1f-e61e80731b21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:16.072820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.072791 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:16.072939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.072822 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:16.072939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.072836 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:16.073957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.073926 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-dshm" pod "89a3994e-d292-4fe4-ba1f-e61e80731b21" (UID: "89a3994e-d292-4fe4-ba1f-e61e80731b21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:16.074031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.074003 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a3994e-d292-4fe4-ba1f-e61e80731b21-tls-certs" pod "89a3994e-d292-4fe4-ba1f-e61e80731b21" (UID: "89a3994e-d292-4fe4-ba1f-e61e80731b21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:15:16.074069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.074045 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a3994e-d292-4fe4-ba1f-e61e80731b21-kube-api-access-pz9tt" pod "89a3994e-d292-4fe4-ba1f-e61e80731b21" (UID: "89a3994e-d292-4fe4-ba1f-e61e80731b21"). InnerVolumeSpecName "kube-api-access-pz9tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:16.174337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.174254 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a3994e-d292-4fe4-ba1f-e61e80731b21-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:16.174337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.174282 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a3994e-d292-4fe4-ba1f-e61e80731b21-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:16.174337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.174292 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9tt\" (UniqueName: \"kubernetes.io/projected/89a3994e-d292-4fe4-ba1f-e61e80731b21-kube-api-access-pz9tt\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:16.308795 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.308757 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg"] Oct 30 01:15:16.900024 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.899991 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" Oct 30 01:15:16.900024 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.899996 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw" event={"ID":"89a3994e-d292-4fe4-ba1f-e61e80731b21","Type":"ContainerDied","Data":"44d842833fd1894e200f2d78d22f70789317ad9b5a0a72463f8169cc1dd98d8c"} Oct 30 01:15:16.900521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:16.900051 4927 scope.go:122] "RemoveContainer" containerID="91fa780c518c4d86ac42de5149d2cda3fdf1d29398eb6f5bb054f384f9213ac4" Oct 30 01:15:17.036594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:17.028685 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw"] Oct 30 01:15:17.059594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:17.058768 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-60e8f4/sub-test-model-60e8f4-kserve-576bc5c555-6znqw"] Oct 30 01:15:17.864895 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:17.864201 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a3994e-d292-4fe4-ba1f-e61e80731b21" path="/var/lib/kubelet/pods/89a3994e-d292-4fe4-ba1f-e61e80731b21/volumes" Oct 30 01:15:19.586597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:19.581463 4927 scope.go:122] "RemoveContainer" containerID="39014976d9a4fab0f39bab351e17f574d81ebd66d0934ef90c8c78e2cd51d621" Oct 30 01:15:20.466621 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:20.459803 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6"] Oct 30 01:15:20.865188 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:20.865107 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:15:21.320597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.319699 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" podUID="37b181ff-a407-4f26-9d90-63131a047c6f" containerName="payload-pre-processing" containerID="cri-o://4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685" gracePeriod=25 Oct 30 01:15:21.755644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.755614 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:15:21.791127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.791091 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b181ff-a407-4f26-9d90-63131a047c6f-kube-api-access-mmk2s\" (UniqueName: \"kubernetes.io/projected/37b181ff-a407-4f26-9d90-63131a047c6f-kube-api-access-mmk2s\") pod \"37b181ff-a407-4f26-9d90-63131a047c6f\" (UID: \"37b181ff-a407-4f26-9d90-63131a047c6f\") " Oct 30 01:15:21.791331 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.791167 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b181ff-a407-4f26-9d90-63131a047c6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b181ff-a407-4f26-9d90-63131a047c6f-plugins-config-volume\") pod \"37b181ff-a407-4f26-9d90-63131a047c6f\" (UID: \"37b181ff-a407-4f26-9d90-63131a047c6f\") " Oct 30 01:15:21.791586 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.791548 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b181ff-a407-4f26-9d90-63131a047c6f-plugins-config-volume" pod "37b181ff-a407-4f26-9d90-63131a047c6f" (UID: "37b181ff-a407-4f26-9d90-63131a047c6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:21.791870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.791850 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b181ff-a407-4f26-9d90-63131a047c6f-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.793281 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.793260 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b181ff-a407-4f26-9d90-63131a047c6f-kube-api-access-mmk2s" pod "37b181ff-a407-4f26-9d90-63131a047c6f" (UID: "37b181ff-a407-4f26-9d90-63131a047c6f"). InnerVolumeSpecName "kube-api-access-mmk2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:21.893045 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.893011 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmk2s\" (UniqueName: \"kubernetes.io/projected/37b181ff-a407-4f26-9d90-63131a047c6f-kube-api-access-mmk2s\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.989600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.989550 4927 generic.go:356] "Generic (PLEG): container finished" podID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerID="ef0dbbbd20e6274666463a8b3a639068f9c9efe3ace4cdfa302d6c6ae0c088f1" exitCode=60 Oct 30 01:15:21.989803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.989599 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" event={"ID":"99c48d2c-90bd-43ca-8b73-70851fca0c4b","Type":"ContainerDied","Data":"ef0dbbbd20e6274666463a8b3a639068f9c9efe3ace4cdfa302d6c6ae0c088f1"} Oct 30 01:15:21.989803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.989694 4927 scope.go:122] "RemoveContainer" containerID="39014976d9a4fab0f39bab351e17f574d81ebd66d0934ef90c8c78e2cd51d621" Oct 30 01:15:21.990152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.990133 4927 scope.go:122] "RemoveContainer" containerID="ef0dbbbd20e6274666463a8b3a639068f9c9efe3ace4cdfa302d6c6ae0c088f1" Oct 30 01:15:21.990632 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:21.990606 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-9lfr9_odh-ai-gateway-infra(99c48d2c-90bd-43ca-8b73-70851fca0c4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" Oct 30 01:15:21.991626 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.991564 4927 generic.go:356] "Generic (PLEG): container finished" podID="37b181ff-a407-4f26-9d90-63131a047c6f" containerID="4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685" exitCode=0 Oct 30 01:15:21.991626 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.991615 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" event={"ID":"37b181ff-a407-4f26-9d90-63131a047c6f","Type":"ContainerDied","Data":"4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685"} Oct 30 01:15:21.991761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.991647 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" Oct 30 01:15:21.991761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:21.991656 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg" event={"ID":"37b181ff-a407-4f26-9d90-63131a047c6f","Type":"ContainerDied","Data":"1b9367f3ed0ef7b5e098d9f923e298c34122fd635b4edb5748e413e6cc9cddbb"} Oct 30 01:15:22.016930 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:22.016901 4927 scope.go:122] "RemoveContainer" containerID="4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685" Oct 30 01:15:22.029521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:22.029475 4927 scope.go:122] "RemoveContainer" containerID="4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685" Oct 30 01:15:22.029941 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:22.029914 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685\": container with ID starting with 4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685 not found: ID does not exist" containerID="4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685" Oct 30 01:15:22.030058 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:22.029963 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685"} err="failed to get container status \"4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685\": rpc error: code = NotFound desc = could not find container \"4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685\": container with ID starting with 4528db07c4d5c86939f3b8dcd83777a840c1d666234701022ebf3b3efc0cd685 not found: ID does not exist" Oct 30 01:15:22.078727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:22.078692 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg"] Oct 30 01:15:22.083716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:22.083686 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-6759f47c47-pzzpg"] Oct 30 01:15:22.563799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:22.563761 4927 scope.go:122] "RemoveContainer" containerID="790597c969f57ea35393808dbe0355b95daeecc10d0a0c7c11ede3a00a8c757c" Oct 30 01:15:22.564149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:22.564124 4927 scope.go:122] "RemoveContainer" containerID="c30b86785b6dd8145facdf4a0b75a30485cc22b710f95e18290cd6f6b33fb6f0" Oct 30 01:15:23.056359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.056318 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9"] Oct 30 01:15:23.248265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.248235 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:23.346755 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.346720 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c48d2c-90bd-43ca-8b73-70851fca0c4b-kube-api-access-ng927\" (UniqueName: \"kubernetes.io/projected/99c48d2c-90bd-43ca-8b73-70851fca0c4b-kube-api-access-ng927\") pod \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\" (UID: \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\") " Oct 30 01:15:23.346936 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.346874 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c48d2c-90bd-43ca-8b73-70851fca0c4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c48d2c-90bd-43ca-8b73-70851fca0c4b-maas-api-service-ca\") pod \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\" (UID: \"99c48d2c-90bd-43ca-8b73-70851fca0c4b\") " Oct 30 01:15:23.347259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.347218 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c48d2c-90bd-43ca-8b73-70851fca0c4b-maas-api-service-ca" pod "99c48d2c-90bd-43ca-8b73-70851fca0c4b" (UID: "99c48d2c-90bd-43ca-8b73-70851fca0c4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:23.347472 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.347456 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c48d2c-90bd-43ca-8b73-70851fca0c4b-maas-api-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:23.349107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.349080 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c48d2c-90bd-43ca-8b73-70851fca0c4b-kube-api-access-ng927" pod "99c48d2c-90bd-43ca-8b73-70851fca0c4b" (UID: "99c48d2c-90bd-43ca-8b73-70851fca0c4b"). InnerVolumeSpecName "kube-api-access-ng927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:23.449230 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.449139 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng927\" (UniqueName: \"kubernetes.io/projected/99c48d2c-90bd-43ca-8b73-70851fca0c4b-kube-api-access-ng927\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:23.552214 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.552170 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f"] Oct 30 01:15:23.553837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.553811 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" Oct 30 01:15:23.554017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554002 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" Oct 30 01:15:23.554187 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554173 4927 cpu_manager.go:414] "containerMap: removing container" podUID="37b181ff-a407-4f26-9d90-63131a047c6f" containerName="payload-pre-processing" Oct 30 01:15:23.554284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554270 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b181ff-a407-4f26-9d90-63131a047c6f" containerName="payload-pre-processing" Oct 30 01:15:23.554384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554372 4927 cpu_manager.go:414] "containerMap: removing container" podUID="89a3994e-d292-4fe4-ba1f-e61e80731b21" containerName="main" Oct 30 01:15:23.554477 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554466 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a3994e-d292-4fe4-ba1f-e61e80731b21" containerName="main" Oct 30 01:15:23.554608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554594 4927 cpu_manager.go:414] "containerMap: removing container" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerName="cleanup" Oct 30 01:15:23.554716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554703 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerName="cleanup" Oct 30 01:15:23.554816 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554805 4927 cpu_manager.go:414] "containerMap: removing container" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerName="cleanup" Oct 30 01:15:23.554901 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554889 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerName="cleanup" Oct 30 01:15:23.555025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.554985 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" Oct 30 01:15:23.555025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555003 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555050 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555058 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555298 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f3ccf6c-514d-45b4-bc56-287f5fd0b9f4" containerName="istio-proxy" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555322 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerName="cleanup" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555335 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3762592f-23e3-476a-a79d-29837a453845" containerName="istio-proxy" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555350 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4316fd2e-adab-4e5e-9b8d-c139100038a4" containerName="istio-proxy" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555372 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b181ff-a407-4f26-9d90-63131a047c6f" containerName="payload-pre-processing" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555384 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a3994e-d292-4fe4-ba1f-e61e80731b21" containerName="main" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555397 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerName="cleanup" Oct 30 01:15:23.556378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.555408 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerName="cleanup" Oct 30 01:15:23.560858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.560830 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.577723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.574937 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f69104\"/\"kube-root-ca.crt\"" Oct 30 01:15:23.577723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.575102 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f69104\"/\"auth-test-model-f69104-kserve-self-signed-certs\"" Oct 30 01:15:23.577723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.575152 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f69104\"/\"openshift-service-ca.crt\"" Oct 30 01:15:23.577723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.575229 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f69104\"/\"default-dockercfg-zkwbp\"" Oct 30 01:15:23.581180 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.581143 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b181ff-a407-4f26-9d90-63131a047c6f" path="/var/lib/kubelet/pods/37b181ff-a407-4f26-9d90-63131a047c6f/volumes" Oct 30 01:15:23.584158 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.584130 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f"] Oct 30 01:15:23.653749 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.653706 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dbz8\" (UniqueName: \"kubernetes.io/projected/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-kube-api-access-7dbz8\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.653971 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.653830 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-home\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.653971 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.653940 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tls-certs\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.654121 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.654079 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-dshm\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.654247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.654221 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-model-cache\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.654410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.654391 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tmp-dir\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.755456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.755421 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-home\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.755653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.755511 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tls-certs\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.755653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.755598 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-dshm\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.755766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.755663 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-model-cache\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.755766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.755738 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tmp-dir\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.755835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.755789 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dbz8\" (UniqueName: \"kubernetes.io/projected/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-kube-api-access-7dbz8\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.755889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.755865 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-home\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.756094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.756065 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-model-cache\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.756224 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.756119 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tmp-dir\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.758044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.758024 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-dshm\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.758239 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.758223 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tls-certs\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.778930 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.778901 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dbz8\" (UniqueName: \"kubernetes.io/projected/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-kube-api-access-7dbz8\") pod \"auth-test-model-f69104-kserve-8565bfc4d5-f6s2f\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:23.884652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:23.884555 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:24.084747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.083032 4927 generic.go:356] "Generic (PLEG): container finished" podID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerID="af41ff9192151470471be6d3ed8bd1e3710fe535327aa9c77fabb528f001b2e5" exitCode=60 Oct 30 01:15:24.084747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.083189 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" event={"ID":"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4","Type":"ContainerDied","Data":"af41ff9192151470471be6d3ed8bd1e3710fe535327aa9c77fabb528f001b2e5"} Oct 30 01:15:24.084747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.083262 4927 scope.go:122] "RemoveContainer" containerID="c30b86785b6dd8145facdf4a0b75a30485cc22b710f95e18290cd6f6b33fb6f0" Oct 30 01:15:24.086319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.086283 4927 scope.go:122] "RemoveContainer" containerID="af41ff9192151470471be6d3ed8bd1e3710fe535327aa9c77fabb528f001b2e5" Oct 30 01:15:24.088181 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:24.088143 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7_odh-ai-gateway-infra(0623ac0b-7aff-435d-9bb9-7031e2ebfcc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" Oct 30 01:15:24.090305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.090058 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" Oct 30 01:15:24.091453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.091425 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9" event={"ID":"99c48d2c-90bd-43ca-8b73-70851fca0c4b","Type":"ContainerDied","Data":"ae1100a2d8ba8e1f8e3b14ebff32a679dd3dcb186df5b9feec4c456d8b5f82da"} Oct 30 01:15:24.102738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.101759 4927 generic.go:356] "Generic (PLEG): container finished" podID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerID="fe703fa8b6695f85473e072eeead9c2642362edabdf52d26724e647f5d862efc" exitCode=60 Oct 30 01:15:24.102738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.101835 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" event={"ID":"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3","Type":"ContainerDied","Data":"fe703fa8b6695f85473e072eeead9c2642362edabdf52d26724e647f5d862efc"} Oct 30 01:15:24.102738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.102252 4927 scope.go:122] "RemoveContainer" containerID="fe703fa8b6695f85473e072eeead9c2642362edabdf52d26724e647f5d862efc" Oct 30 01:15:24.102738 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:24.102639 4927 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2_odh-ai-gateway-infra(fd6cc7a3-70b9-439a-918a-aa92fd89a5b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" Oct 30 01:15:24.111441 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.111400 4927 scope.go:122] "RemoveContainer" containerID="ef0dbbbd20e6274666463a8b3a639068f9c9efe3ace4cdfa302d6c6ae0c088f1" Oct 30 01:15:24.173485 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.173435 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9"] Oct 30 01:15:24.178281 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.177936 4927 scope.go:122] "RemoveContainer" containerID="790597c969f57ea35393808dbe0355b95daeecc10d0a0c7c11ede3a00a8c757c" Oct 30 01:15:24.181186 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.181158 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-9lfr9"] Oct 30 01:15:24.398485 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.398447 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f"] Oct 30 01:15:24.478113 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.478084 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z"] Oct 30 01:15:24.479149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.479125 4927 cpu_manager.go:414] "containerMap: removing container" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerName="cleanup" Oct 30 01:15:24.479149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.479149 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" containerName="cleanup" Oct 30 01:15:24.483594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.483558 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.499038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.499008 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f91e87-openshift-default-dockercfg-6k6mt\"" Oct 30 01:15:24.506552 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.506523 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z"] Oct 30 01:15:24.571493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.571440 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c66b95-acdf-4fbf-997f-01e779dac926-istiod-ca-cert\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.571493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.571492 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c66b95-acdf-4fbf-997f-01e779dac926-istio-podinfo\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.571867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.571516 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.571867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.571585 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttv9t\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-kube-api-access-ttv9t\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.571867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.571632 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-envoy\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.571867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.571763 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-credential-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.572098 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.571958 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-istio-token\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.572098 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.571996 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-certs\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.572098 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.572062 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-data\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.674127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.674007 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-istio-token\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.674127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.674072 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-certs\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.674392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.674168 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-data\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.674392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.674212 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c66b95-acdf-4fbf-997f-01e779dac926-istiod-ca-cert\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.674392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.674245 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c66b95-acdf-4fbf-997f-01e779dac926-istio-podinfo\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.674392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.674275 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.674392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.674348 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttv9t\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-kube-api-access-ttv9t\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.674727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.674393 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-envoy\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.674727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.674593 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-credential-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.675305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.675276 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-credential-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.675825 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.675800 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.676018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.675993 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-data\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.676436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.676408 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-certs\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.676787 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.676742 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c66b95-acdf-4fbf-997f-01e779dac926-istiod-ca-cert\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.678936 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.678908 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-envoy\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.679310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.679287 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c66b95-acdf-4fbf-997f-01e779dac926-istio-podinfo\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.702021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.701980 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-istio-token\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.702202 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.702111 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttv9t\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-kube-api-access-ttv9t\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:24.801623 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:24.801584 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:25.120313 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.120261 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" event={"ID":"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c","Type":"ContainerStarted","Data":"573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407"} Oct 30 01:15:25.120726 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.120313 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" event={"ID":"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c","Type":"ContainerStarted","Data":"c2312781b1bb687e85fc6c1d6cf30b353276ea92b0c1168fe053fde206d2102d"} Oct 30 01:15:25.219811 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.218118 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:25.219811 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.218227 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:25.219811 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.218263 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:25.251747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.251663 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7"] Oct 30 01:15:25.260492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.260461 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z"] Oct 30 01:15:25.270644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.270528 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" podStartSLOduration=2.270511014 podStartE2EDuration="2.270511014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:25.17589399 +0000 UTC m=+2463.747820215" watchObservedRunningTime="2026-10-30 01:15:25.270511014 +0000 UTC m=+2463.842437237" Oct 30 01:15:25.281457 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.281422 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2"] Oct 30 01:15:25.557225 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.557123 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:25.584596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.581675 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c48d2c-90bd-43ca-8b73-70851fca0c4b" path="/var/lib/kubelet/pods/99c48d2c-90bd-43ca-8b73-70851fca0c4b/volumes" Oct 30 01:15:25.655326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.655291 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-kube-api-access-h74bv\" (UniqueName: \"kubernetes.io/projected/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-kube-api-access-h74bv\") pod \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\" (UID: \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\") " Oct 30 01:15:25.655556 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.655538 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-maas-api-service-ca\") pod \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\" (UID: \"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4\") " Oct 30 01:15:25.655870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.655845 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:25.657358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.657331 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-maas-api-service-ca" pod "0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" (UID: "0623ac0b-7aff-435d-9bb9-7031e2ebfcc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:25.659457 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.659403 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-kube-api-access-h74bv" pod "0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" (UID: "0623ac0b-7aff-435d-9bb9-7031e2ebfcc4"). InnerVolumeSpecName "kube-api-access-h74bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:25.758232 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.758194 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-maas-api-service-ca\") pod \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\" (UID: \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\") " Oct 30 01:15:25.758627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.758596 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-kube-api-access-jnsfl\" (UniqueName: \"kubernetes.io/projected/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-kube-api-access-jnsfl\") pod \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\" (UID: \"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3\") " Oct 30 01:15:25.758772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.758622 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-maas-api-service-ca" pod "fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" (UID: "fd6cc7a3-70b9-439a-918a-aa92fd89a5b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:25.759695 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.759675 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-maas-api-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:25.759695 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.759696 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-maas-api-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:25.759883 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.759707 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h74bv\" (UniqueName: \"kubernetes.io/projected/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4-kube-api-access-h74bv\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:25.761423 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.761402 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-kube-api-access-jnsfl" pod "fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" (UID: "fd6cc7a3-70b9-439a-918a-aa92fd89a5b3"). InnerVolumeSpecName "kube-api-access-jnsfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:25.861661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.861622 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsfl\" (UniqueName: \"kubernetes.io/projected/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3-kube-api-access-jnsfl\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:25.885662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:25.885622 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:26.134979 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.134944 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" Oct 30 01:15:26.135491 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.134954 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2" event={"ID":"fd6cc7a3-70b9-439a-918a-aa92fd89a5b3","Type":"ContainerDied","Data":"a1dce0b1c13603fa50730e9aa7bff6b81d4576f2353c5da578cb9cefe158c9db"} Oct 30 01:15:26.135491 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.135071 4927 scope.go:122] "RemoveContainer" containerID="fe703fa8b6695f85473e072eeead9c2642362edabdf52d26724e647f5d862efc" Oct 30 01:15:26.137075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.137046 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" event={"ID":"05c66b95-acdf-4fbf-997f-01e779dac926","Type":"ContainerStarted","Data":"9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff"} Oct 30 01:15:26.137226 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.137107 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" event={"ID":"05c66b95-acdf-4fbf-997f-01e779dac926","Type":"ContainerStarted","Data":"3686e58780aaddd5442ac0279b4c700207154295b329efaa68b63ba9a4908f43"} Oct 30 01:15:26.138770 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.138744 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" Oct 30 01:15:26.138942 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.138910 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7" event={"ID":"0623ac0b-7aff-435d-9bb9-7031e2ebfcc4","Type":"ContainerDied","Data":"d094950072852d8eb96f39dee1037a94c762fb93effdccf38b76ebbfccd1e32e"} Oct 30 01:15:26.149992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.149965 4927 scope.go:122] "RemoveContainer" containerID="af41ff9192151470471be6d3ed8bd1e3710fe535327aa9c77fabb528f001b2e5" Oct 30 01:15:26.195392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.195329 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" podStartSLOduration=2.195312689 podStartE2EDuration="2.195312689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:26.193710137 +0000 UTC m=+2464.765636360" watchObservedRunningTime="2026-10-30 01:15:26.195312689 +0000 UTC m=+2464.767238911" Oct 30 01:15:26.233154 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.233113 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2"] Oct 30 01:15:26.240683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.240654 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888715-h4tv2"] Oct 30 01:15:26.272001 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.271961 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7"] Oct 30 01:15:26.276752 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.276722 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888715-k9dr7"] Oct 30 01:15:26.802598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.802547 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:26.805123 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.805083 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:15:26.805283 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:26.805133 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 01:15:27.585596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:27.574745 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" path="/var/lib/kubelet/pods/0623ac0b-7aff-435d-9bb9-7031e2ebfcc4/volumes" Oct 30 01:15:27.598596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:27.596645 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" path="/var/lib/kubelet/pods/fd6cc7a3-70b9-439a-918a-aa92fd89a5b3/volumes" Oct 30 01:15:27.802297 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:27.802256 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:15:27.802489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:27.802322 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 01:15:28.807592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:28.803146 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:15:28.807592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:28.803228 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 01:15:29.807640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:29.806791 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:15:29.807640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:29.806877 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 01:15:30.805728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:30.805687 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:15:30.805963 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:30.805762 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 01:15:31.802428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:31.802386 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 01:15:31.802912 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:31.802458 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 01:15:32.807639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:32.807533 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:32.808096 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:32.808062 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:32.808933 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:32.808909 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:15:33.888973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:33.888933 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:33.904012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:33.903974 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:34.259813 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:34.259741 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:15:34.450229 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:34.450192 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:15:35.987302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:35.987258 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" podUID="160deffa-69cd-4ef7-80e8-4aa6112e0b9d" containerName="main" containerID="cri-o://9a3ace08c686576e00d6ae53009d414dbdc408853b33c366fe650ff6dd2993df" gracePeriod=45 Oct 30 01:15:36.268913 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.268878 4927 generic.go:356] "Generic (PLEG): container finished" podID="160deffa-69cd-4ef7-80e8-4aa6112e0b9d" containerID="9a3ace08c686576e00d6ae53009d414dbdc408853b33c366fe650ff6dd2993df" exitCode=0 Oct 30 01:15:36.269057 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.268962 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" event={"ID":"160deffa-69cd-4ef7-80e8-4aa6112e0b9d","Type":"ContainerDied","Data":"9a3ace08c686576e00d6ae53009d414dbdc408853b33c366fe650ff6dd2993df"} Oct 30 01:15:36.269057 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.269006 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" event={"ID":"160deffa-69cd-4ef7-80e8-4aa6112e0b9d","Type":"ContainerDied","Data":"d9383c5f365b747adcf8fc814231d23c2a5533368069eb34eb0287c5e10699e3"} Oct 30 01:15:36.269057 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.269022 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9383c5f365b747adcf8fc814231d23c2a5533368069eb34eb0287c5e10699e3" Oct 30 01:15:36.293981 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.293952 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:15:36.324498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.324470 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tmp-dir\") pod \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " Oct 30 01:15:36.324669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.324557 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-model-cache\") pod \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " Oct 30 01:15:36.324669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.324656 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-kube-api-access-9kmhb\" (UniqueName: \"kubernetes.io/projected/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-kube-api-access-9kmhb\") pod \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " Oct 30 01:15:36.324816 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.324708 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tls-certs\" (UniqueName: \"kubernetes.io/secret/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tls-certs\") pod \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " Oct 30 01:15:36.324816 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.324779 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-dshm\") pod \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " Oct 30 01:15:36.324816 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.324770 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tmp-dir" pod "160deffa-69cd-4ef7-80e8-4aa6112e0b9d" (UID: "160deffa-69cd-4ef7-80e8-4aa6112e0b9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:36.324975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.324805 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-home\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-home\") pod \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\" (UID: \"160deffa-69cd-4ef7-80e8-4aa6112e0b9d\") " Oct 30 01:15:36.324975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.324851 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-model-cache" pod "160deffa-69cd-4ef7-80e8-4aa6112e0b9d" (UID: "160deffa-69cd-4ef7-80e8-4aa6112e0b9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:36.325054 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.325023 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-home" pod "160deffa-69cd-4ef7-80e8-4aa6112e0b9d" (UID: "160deffa-69cd-4ef7-80e8-4aa6112e0b9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:36.325776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.325740 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:36.325776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.325762 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:36.325776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.325775 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:36.327042 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.327011 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-dshm" pod "160deffa-69cd-4ef7-80e8-4aa6112e0b9d" (UID: "160deffa-69cd-4ef7-80e8-4aa6112e0b9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:36.327136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.327067 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-kube-api-access-9kmhb" pod "160deffa-69cd-4ef7-80e8-4aa6112e0b9d" (UID: "160deffa-69cd-4ef7-80e8-4aa6112e0b9d"). InnerVolumeSpecName "kube-api-access-9kmhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:36.327136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.327091 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tls-certs" pod "160deffa-69cd-4ef7-80e8-4aa6112e0b9d" (UID: "160deffa-69cd-4ef7-80e8-4aa6112e0b9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:15:36.427704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.427668 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kmhb\" (UniqueName: \"kubernetes.io/projected/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-kube-api-access-9kmhb\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:36.427704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.427699 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:36.427704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:36.427710 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160deffa-69cd-4ef7-80e8-4aa6112e0b9d-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:37.277168 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:37.277140 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6" Oct 30 01:15:37.331033 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:37.330991 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6"] Oct 30 01:15:37.340808 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:37.340778 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/auth-test-model-6c4f6f-kserve-75d88949c-p2hb6"] Oct 30 01:15:37.345999 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:37.345969 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod160deffa_69cd_4ef7_80e8_4aa6112e0b9d.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:37.574893 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:37.574810 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160deffa-69cd-4ef7-80e8-4aa6112e0b9d" path="/var/lib/kubelet/pods/160deffa-69cd-4ef7-80e8-4aa6112e0b9d/volumes" Oct 30 01:15:39.872750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.872713 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z"] Oct 30 01:15:39.874870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874843 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerName="cleanup" Oct 30 01:15:39.874870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874868 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerName="cleanup" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874883 4927 cpu_manager.go:414] "containerMap: removing container" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerName="cleanup" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874889 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerName="cleanup" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874902 4927 cpu_manager.go:414] "containerMap: removing container" podUID="160deffa-69cd-4ef7-80e8-4aa6112e0b9d" containerName="main" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874908 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160deffa-69cd-4ef7-80e8-4aa6112e0b9d" containerName="main" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874920 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerName="cleanup" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874925 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerName="cleanup" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874936 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerName="cleanup" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874940 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerName="cleanup" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874963 4927 cpu_manager.go:414] "containerMap: removing container" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerName="cleanup" Oct 30 01:15:39.875025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.874967 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerName="cleanup" Oct 30 01:15:39.875359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.875110 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerName="cleanup" Oct 30 01:15:39.875359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.875120 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerName="cleanup" Oct 30 01:15:39.875359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.875134 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerName="cleanup" Oct 30 01:15:39.875359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.875142 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="160deffa-69cd-4ef7-80e8-4aa6112e0b9d" containerName="main" Oct 30 01:15:39.875359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.875148 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerName="cleanup" Oct 30 01:15:39.875359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.875155 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0623ac0b-7aff-435d-9bb9-7031e2ebfcc4" containerName="cleanup" Oct 30 01:15:39.880909 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.880887 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:39.910667 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.909679 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-60e8f4-openshift-default-dockercfg-665t8\"" Oct 30 01:15:39.918550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.918512 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z"] Oct 30 01:15:39.987536 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.987495 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:39.987784 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.987549 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:39.987784 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.987733 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:39.987923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.987821 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:39.987923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.987856 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zj94\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-kube-api-access-2zj94\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:39.987923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.987884 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:39.988066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.987973 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:39.988066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.988045 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b5a9b-2a51-43fe-9104-f7a09680fa16-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:39.988150 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:39.988101 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.089693 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.089658 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.089693 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.089705 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.089965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.089745 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.089965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.089787 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.089965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.089808 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zj94\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-kube-api-access-2zj94\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.089965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.089834 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.089965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.089885 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.090262 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.089981 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b5a9b-2a51-43fe-9104-f7a09680fa16-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.090262 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.090062 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.090373 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.090339 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.090462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.090420 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.090462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.090440 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.090661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.090641 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.090842 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.090817 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b5a9b-2a51-43fe-9104-f7a09680fa16-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.092228 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.092206 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.092846 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.092830 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.112400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.112364 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.112585 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.112493 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zj94\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-kube-api-access-2zj94\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.198747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.198707 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:40.545587 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.545406 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999"] Oct 30 01:15:40.546690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.546660 4927 cpu_manager.go:414] "containerMap: removing container" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerName="cleanup" Oct 30 01:15:40.546690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.546688 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerName="cleanup" Oct 30 01:15:40.546962 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.546920 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6cc7a3-70b9-439a-918a-aa92fd89a5b3" containerName="cleanup" Oct 30 01:15:40.548823 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.548789 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:40.548933 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.548868 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:40.548933 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.548910 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:40.552879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.552859 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z"] Oct 30 01:15:40.552986 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.552969 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.568954 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.568921 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999"] Oct 30 01:15:40.569880 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.569853 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-abf8ae-serving-cert\"" Oct 30 01:15:40.603603 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.603487 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04c018e-d268-4c73-97c2-63436cebfd55-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.603603 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.603600 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzf2\" (UniqueName: \"kubernetes.io/projected/b04c018e-d268-4c73-97c2-63436cebfd55-kube-api-access-wvzf2\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.603922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.603820 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-metrics-certs\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.605548 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.604149 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.766595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.761681 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-metrics-certs\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.766595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.761882 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.766595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.762291 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04c018e-d268-4c73-97c2-63436cebfd55-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.766595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.762400 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzf2\" (UniqueName: \"kubernetes.io/projected/b04c018e-d268-4c73-97c2-63436cebfd55-kube-api-access-wvzf2\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.797620 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.797470 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-metrics-certs\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.810954 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.810893 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04c018e-d268-4c73-97c2-63436cebfd55-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.828639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.825220 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.828639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.826327 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzf2\" (UniqueName: \"kubernetes.io/projected/b04c018e-d268-4c73-97c2-63436cebfd55-kube-api-access-wvzf2\") pod \"maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:40.880287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:40.880246 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:41.197453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:41.197423 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999"] Oct 30 01:15:41.198992 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:15:41.198961 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04c018e_d268_4c73_97c2_63436cebfd55.slice/crio-7ec17d7bec41101b81c51ed77685b0dee7584b1afc995d20844146af4cf087bb WatchSource:0}: Error finding container 7ec17d7bec41101b81c51ed77685b0dee7584b1afc995d20844146af4cf087bb: Status 404 returned error can't find the container with id 7ec17d7bec41101b81c51ed77685b0dee7584b1afc995d20844146af4cf087bb Oct 30 01:15:41.320185 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:41.320143 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" event={"ID":"457b5a9b-2a51-43fe-9104-f7a09680fa16","Type":"ContainerStarted","Data":"13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411"} Oct 30 01:15:41.320402 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:41.320212 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" event={"ID":"457b5a9b-2a51-43fe-9104-f7a09680fa16","Type":"ContainerStarted","Data":"e822740c1561401f0e1f0aae585107209b5a2ed1a3fdb933b7a5aca50d53b2ba"} Oct 30 01:15:41.321867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:41.321833 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" event={"ID":"b04c018e-d268-4c73-97c2-63436cebfd55","Type":"ContainerStarted","Data":"7ec17d7bec41101b81c51ed77685b0dee7584b1afc995d20844146af4cf087bb"} Oct 30 01:15:41.366044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:41.365987 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" podStartSLOduration=2.3659733960000002 podStartE2EDuration="2.365973396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:41.362331054 +0000 UTC m=+2479.934257275" watchObservedRunningTime="2026-10-30 01:15:41.365973396 +0000 UTC m=+2479.937899617" Oct 30 01:15:42.199993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:42.199918 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:42.217126 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:42.217076 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 01:15:42.217482 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:42.217437 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 01:15:42.333911 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:42.333863 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" event={"ID":"b04c018e-d268-4c73-97c2-63436cebfd55","Type":"ContainerStarted","Data":"ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df"} Oct 30 01:15:42.334120 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:42.333948 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:42.385593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:42.374069 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" podStartSLOduration=1.946148092 podStartE2EDuration="2.374049331s" totalImagesPullingTime="427.901239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:42.373749143 +0000 UTC m=+2480.945675366" watchObservedRunningTime="2026-10-30 01:15:42.374049331 +0000 UTC m=+2480.945975556" Oct 30 01:15:43.200920 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:43.200874 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 01:15:43.201464 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:43.200948 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 01:15:44.199973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:44.199936 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 01:15:44.200208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:44.200000 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 01:15:45.200096 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:45.200050 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 01:15:45.200696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:45.200112 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 01:15:46.205654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:46.205610 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 01:15:46.206315 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:46.206275 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 01:15:47.380079 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:47.380044 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:47.382652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:47.381077 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:47.384828 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:47.384808 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:15:48.348673 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:48.348636 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:15:52.951009 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:52.946217 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj"] Oct 30 01:15:52.951009 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:52.946428 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" containerID="cri-o://76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa" gracePeriod=30 Oct 30 01:15:57.048219 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:57.048181 4927 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:15:57.048977 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:57.048935 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:15:57.967308 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:57.967266 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m"] Oct 30 01:15:57.974621 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:57.973973 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:57.997714 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:57.997667 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m"] Oct 30 01:15:57.999035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:57.998996 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c4f6f\"/\"kube-root-ca.crt\"" Oct 30 01:15:58.005842 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.005815 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c4f6f\"/\"rate-test-model-6c4f6f-kserve-self-signed-certs\"" Oct 30 01:15:58.005949 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.005819 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c4f6f\"/\"openshift-service-ca.crt\"" Oct 30 01:15:58.005949 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.005825 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c4f6f\"/\"default-dockercfg-f7q9v\"" Oct 30 01:15:58.073238 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.073205 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tls-certs\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.073636 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.073355 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tmp-dir\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.073636 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.073551 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-model-cache\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.074017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.073726 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-home\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.074017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.073764 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9557\" (UniqueName: \"kubernetes.io/projected/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-kube-api-access-p9557\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.074017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.073799 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-dshm\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.176171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.175813 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-home\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.176171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.175904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9557\" (UniqueName: \"kubernetes.io/projected/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-kube-api-access-p9557\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.176171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.175958 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-dshm\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.176171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.176037 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tls-certs\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.176171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.176160 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tmp-dir\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.176639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.176308 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-model-cache\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.177908 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.177041 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-home\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.177908 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.177783 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tmp-dir\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.177908 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.177843 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-model-cache\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.184386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.182985 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-dshm\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.184942 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.184918 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tls-certs\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.206124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.206092 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9557\" (UniqueName: \"kubernetes.io/projected/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-kube-api-access-p9557\") pod \"rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.272359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.272326 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 01:15:58.364060 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.364023 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:15:58.379880 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.379850 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-credential-socket\") pod \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " Oct 30 01:15:58.380061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.379928 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-socket\") pod \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " Oct 30 01:15:58.380061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.379977 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-certs\") pod \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " Oct 30 01:15:58.380061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.380033 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-kube-api-access-lmkrn\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-kube-api-access-lmkrn\") pod \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " Oct 30 01:15:58.380239 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.380141 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-token\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-token\") pod \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " Oct 30 01:15:58.380239 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.380152 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-credential-socket" pod "c1e70cbf-7f1b-4507-81ce-a88fff91e343" (UID: "c1e70cbf-7f1b-4507-81ce-a88fff91e343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:58.380699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.380390 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-envoy\") pod \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " Oct 30 01:15:58.380699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.380455 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istiod-ca-cert\") pod \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " Oct 30 01:15:58.380699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.380505 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-data\") pod \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " Oct 30 01:15:58.380699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.380655 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-podinfo\") pod \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\" (UID: \"c1e70cbf-7f1b-4507-81ce-a88fff91e343\") " Oct 30 01:15:58.381031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.380972 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-data" pod "c1e70cbf-7f1b-4507-81ce-a88fff91e343" (UID: "c1e70cbf-7f1b-4507-81ce-a88fff91e343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:58.381696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.381666 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:58.381805 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.381702 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:58.381926 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.381887 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istiod-ca-cert" pod "c1e70cbf-7f1b-4507-81ce-a88fff91e343" (UID: "c1e70cbf-7f1b-4507-81ce-a88fff91e343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:58.382135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.382101 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-certs" pod "c1e70cbf-7f1b-4507-81ce-a88fff91e343" (UID: "c1e70cbf-7f1b-4507-81ce-a88fff91e343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:58.382539 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.382514 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-socket" pod "c1e70cbf-7f1b-4507-81ce-a88fff91e343" (UID: "c1e70cbf-7f1b-4507-81ce-a88fff91e343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:58.383286 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.383260 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-token" pod "c1e70cbf-7f1b-4507-81ce-a88fff91e343" (UID: "c1e70cbf-7f1b-4507-81ce-a88fff91e343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:58.383669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.383643 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-podinfo" pod "c1e70cbf-7f1b-4507-81ce-a88fff91e343" (UID: "c1e70cbf-7f1b-4507-81ce-a88fff91e343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:15:58.384325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.384295 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-envoy" pod "c1e70cbf-7f1b-4507-81ce-a88fff91e343" (UID: "c1e70cbf-7f1b-4507-81ce-a88fff91e343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:58.385334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.385313 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-kube-api-access-lmkrn" pod "c1e70cbf-7f1b-4507-81ce-a88fff91e343" (UID: "c1e70cbf-7f1b-4507-81ce-a88fff91e343"). InnerVolumeSpecName "kube-api-access-lmkrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:58.483761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.483672 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:58.483761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.483709 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:58.483761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.483723 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmkrn\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-kube-api-access-lmkrn\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:58.483761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.483738 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:58.483761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.483752 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:58.483761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.483764 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:58.484159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.483777 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e70cbf-7f1b-4507-81ce-a88fff91e343-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:15:58.751755 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.751653 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w"] Oct 30 01:15:58.828617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.825678 4927 generic.go:356] "Generic (PLEG): container finished" podID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerID="76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa" exitCode=0 Oct 30 01:15:58.828617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.825931 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" Oct 30 01:15:58.886630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.882267 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" Oct 30 01:15:58.886630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.882301 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" Oct 30 01:15:58.886630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.882722 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" containerName="istio-proxy" Oct 30 01:15:58.952748 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:15:58.952698 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod008ecceb_3fa1_4cb8_8c30_74eafaa4f7ed.slice/crio-7ad7309911d99afa74d41354f9ed5fcb19d32cb2f359657acc551ba45d49db53 WatchSource:0}: Error finding container 7ad7309911d99afa74d41354f9ed5fcb19d32cb2f359657acc551ba45d49db53: Status 404 returned error can't find the container with id 7ad7309911d99afa74d41354f9ed5fcb19d32cb2f359657acc551ba45d49db53 Oct 30 01:15:58.953809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.953240 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" event={"ID":"c1e70cbf-7f1b-4507-81ce-a88fff91e343","Type":"ContainerDied","Data":"76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa"} Oct 30 01:15:58.953809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.953305 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w"] Oct 30 01:15:58.953809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.953370 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj" event={"ID":"c1e70cbf-7f1b-4507-81ce-a88fff91e343","Type":"ContainerDied","Data":"e15be7a28797aa509b60a992b44958edf116cb318ecca2aa624d88ec69710b12"} Oct 30 01:15:58.953809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.953393 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8"] Oct 30 01:15:58.953809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.953392 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:15:58.953809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.953453 4927 scope.go:122] "RemoveContainer" containerID="76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa" Oct 30 01:15:58.966947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.965928 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m"] Oct 30 01:15:58.966947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.966138 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:15:58.970282 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.970233 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f91e87-dockercfg-dwpwk\"" Oct 30 01:15:58.970282 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.970264 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f91e87\"" Oct 30 01:15:59.001337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.988351 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrw7z\" (UniqueName: \"kubernetes.io/projected/6d2cff76-9eb0-4def-aba6-0a033eaecd42-kube-api-access-jrw7z\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w\" (UID: \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:15:59.001337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:58.988592 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2cff76-9eb0-4def-aba6-0a033eaecd42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w\" (UID: \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:15:59.015719 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.012902 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8"] Oct 30 01:15:59.027548 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.023701 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj"] Oct 30 01:15:59.034884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.033195 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kjkfj"] Oct 30 01:15:59.047188 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.047073 4927 scope.go:122] "RemoveContainer" containerID="76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa" Oct 30 01:15:59.048188 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:15:59.048096 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa\": container with ID starting with 76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa not found: ID does not exist" containerID="76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa" Oct 30 01:15:59.048188 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.048135 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa"} err="failed to get container status \"76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa\": rpc error: code = NotFound desc = could not find container \"76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa\": container with ID starting with 76a03e2c3cec9d230bc68d7ea520d7be2895271e700316ac9269d1016e3006fa not found: ID does not exist" Oct 30 01:15:59.089909 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.089824 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrw7z\" (UniqueName: \"kubernetes.io/projected/6d2cff76-9eb0-4def-aba6-0a033eaecd42-kube-api-access-jrw7z\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w\" (UID: \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:15:59.090366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.090031 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2cff76-9eb0-4def-aba6-0a033eaecd42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w\" (UID: \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:15:59.090366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.090130 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8\" (UID: \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:15:59.090366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.090180 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkszq\" (UniqueName: \"kubernetes.io/projected/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-kube-api-access-qkszq\") pod \"payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8\" (UID: \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:15:59.092271 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.092224 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2cff76-9eb0-4def-aba6-0a033eaecd42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w\" (UID: \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:15:59.118307 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.118247 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrw7z\" (UniqueName: \"kubernetes.io/projected/6d2cff76-9eb0-4def-aba6-0a033eaecd42-kube-api-access-jrw7z\") pod \"payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w\" (UID: \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:15:59.192514 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.192467 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkszq\" (UniqueName: \"kubernetes.io/projected/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-kube-api-access-qkszq\") pod \"payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8\" (UID: \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:15:59.192946 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.192926 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8\" (UID: \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:15:59.193703 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.193675 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8\" (UID: \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:15:59.215083 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.215056 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkszq\" (UniqueName: \"kubernetes.io/projected/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-kube-api-access-qkszq\") pod \"payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8\" (UID: \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:15:59.303228 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.303147 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:15:59.361668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.361623 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:15:59.579513 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.579240 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e70cbf-7f1b-4507-81ce-a88fff91e343" path="/var/lib/kubelet/pods/c1e70cbf-7f1b-4507-81ce-a88fff91e343/volumes" Oct 30 01:15:59.626056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.626028 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w"] Oct 30 01:15:59.626189 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:15:59.626029 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2cff76_9eb0_4def_aba6_0a033eaecd42.slice/crio-75453d7eeab2369698237690a844b6409bb461eebae0438ac0f0b320a7abfaea WatchSource:0}: Error finding container 75453d7eeab2369698237690a844b6409bb461eebae0438ac0f0b320a7abfaea: Status 404 returned error can't find the container with id 75453d7eeab2369698237690a844b6409bb461eebae0438ac0f0b320a7abfaea Oct 30 01:15:59.684614 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:15:59.684561 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636f26b2_b3ba_46d6_95fb_b49ad3b9315e.slice/crio-7a40ac70e578c8fa6701b8640d6a3cd45ff07741329edaa30bd838be5e6de6b5 WatchSource:0}: Error finding container 7a40ac70e578c8fa6701b8640d6a3cd45ff07741329edaa30bd838be5e6de6b5: Status 404 returned error can't find the container with id 7a40ac70e578c8fa6701b8640d6a3cd45ff07741329edaa30bd838be5e6de6b5 Oct 30 01:15:59.686463 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.686434 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8"] Oct 30 01:15:59.836214 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.836104 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" event={"ID":"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed","Type":"ContainerStarted","Data":"c610d9757707a4d806493798b38e852437825f41143a3a228a8457e8ef6671a8"} Oct 30 01:15:59.836214 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.836175 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" event={"ID":"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed","Type":"ContainerStarted","Data":"7ad7309911d99afa74d41354f9ed5fcb19d32cb2f359657acc551ba45d49db53"} Oct 30 01:15:59.837717 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.837690 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" event={"ID":"636f26b2-b3ba-46d6-95fb-b49ad3b9315e","Type":"ContainerStarted","Data":"7a40ac70e578c8fa6701b8640d6a3cd45ff07741329edaa30bd838be5e6de6b5"} Oct 30 01:15:59.841027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.841003 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" event={"ID":"6d2cff76-9eb0-4def-aba6-0a033eaecd42","Type":"ContainerStarted","Data":"75453d7eeab2369698237690a844b6409bb461eebae0438ac0f0b320a7abfaea"} Oct 30 01:15:59.879791 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:15:59.879741 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" podStartSLOduration=2.879726523 podStartE2EDuration="2.879726523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:59.875477728 +0000 UTC m=+2498.447403942" watchObservedRunningTime="2026-10-30 01:15:59.879726523 +0000 UTC m=+2498.451652744" Oct 30 01:16:00.365014 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:00.364944 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:16:01.864725 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:01.864629 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" event={"ID":"636f26b2-b3ba-46d6-95fb-b49ad3b9315e","Type":"ContainerStarted","Data":"7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e"} Oct 30 01:16:01.865155 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:01.864759 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:16:01.866834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:01.866801 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" event={"ID":"6d2cff76-9eb0-4def-aba6-0a033eaecd42","Type":"ContainerStarted","Data":"46ea7c73452e558247db57c4748c5e1c20673808dda7de9c26c7e96dc64f333b"} Oct 30 01:16:01.866989 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:01.866863 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:16:01.909367 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:01.909309 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" podStartSLOduration=1.972723722 podStartE2EDuration="3.909295336s" totalImagesPullingTime="1.936571614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:01.906509596 +0000 UTC m=+2500.478435828" watchObservedRunningTime="2026-10-30 01:16:01.909295336 +0000 UTC m=+2500.481221604" Oct 30 01:16:01.961240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:01.961169 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" podStartSLOduration=1.979788844 podStartE2EDuration="3.961150028s" totalImagesPullingTime="1.981361184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:01.95519295 +0000 UTC m=+2500.527119173" watchObservedRunningTime="2026-10-30 01:16:01.961150028 +0000 UTC m=+2500.533076251" Oct 30 01:16:04.444756 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:16:04.444718 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:16:08.365040 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:08.365006 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:16:08.377467 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:08.377433 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:16:08.940689 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:08.940655 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:16:12.881605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:12.881517 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:16:12.906592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:12.901845 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:16:14.249774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:14.249715 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" podUID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 01:16:14.257612 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:14.250271 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.378654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:14.363682 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.378654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:14.367667 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" podUID="7524c2d1-a405-4f4b-90ff-ffd280dd1840" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/health\": context deadline exceeded" Oct 30 01:16:14.532601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:14.531730 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:14.560405 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:14.559658 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 01:16:15.097669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.094022 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" podUID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/health\": context deadline exceeded" Oct 30 01:16:15.253605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.200588 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 01:16:15.254316 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.254261 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4" podUID="1aedaf3b-4fbd-485d-bf58-319230ad826a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 01:16:15.319764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.319720 4927 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded" Oct 30 01:16:15.442736 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.386884 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" podUID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 01:16:15.478624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.397683 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.497407 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.497369 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:16:15.497724 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.497701 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 01:16:15.498226 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.429917 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 01:16:15.561615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.552889 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:16:15.561952 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.561916 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 01:16:15.620834 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.619289 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" podUID="008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:15.835662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.826851 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" podUID="97acea83-49aa-4c90-bf1c-312dc1dd7d02" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.118:50052\" within 1s: context deadline exceeded" Oct 30 01:16:15.848732 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:16:15.848705 4927 logging.go:55] [core] [Channel #10349 SubChannel #10350]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.118:50052", ServerName: "10.132.0.118:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.118:50052: operation was canceled" Oct 30 01:16:15.870589 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:15.864257 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 01:16:16.025784 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:16.025758 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 01:16:24.793269 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:24.791953 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": read tcp 10.132.0.2:34876->10.132.0.188:15021: read: connection reset by peer" start-of-body= Oct 30 01:16:24.793269 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:24.792027 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": read tcp 10.132.0.2:34876->10.132.0.188:15021: read: connection reset by peer" Oct 30 01:16:24.897494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:24.897456 4927 generic.go:356] "Generic (PLEG): container finished" podID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerID="7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74" exitCode=0 Oct 30 01:16:24.897708 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:24.897529 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" event={"ID":"28e21533-1c44-41fa-82a3-f5ed8296bdb4","Type":"ContainerDied","Data":"7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74"} Oct 30 01:16:24.897979 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:24.897962 4927 scope.go:122] "RemoveContainer" containerID="7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74" Oct 30 01:16:24.900031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:24.899997 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:16:24.900106 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:24.900093 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:16:24.900152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:24.900128 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:16:25.914145 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:25.914104 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" event={"ID":"28e21533-1c44-41fa-82a3-f5ed8296bdb4","Type":"ContainerStarted","Data":"667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70"} Oct 30 01:16:26.246680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:26.246596 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:16:26.246839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:26.246776 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:16:26.246839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:26.246817 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:16:27.246094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.246052 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:16:27.246500 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.246113 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:16:27.753201 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.753165 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk"] Oct 30 01:16:27.761033 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.761006 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.792134 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.792095 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk"] Oct 30 01:16:27.808856 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.808806 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-33e759\"/\"kube-root-ca.crt\"" Oct 30 01:16:27.809219 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.809193 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-33e759\"/\"default-dockercfg-6qxp9\"" Oct 30 01:16:27.820128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.820092 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-33e759\"/\"rate-test-model-33e759-kserve-self-signed-certs\"" Oct 30 01:16:27.820260 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.820156 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-33e759\"/\"openshift-service-ca.crt\"" Oct 30 01:16:27.865977 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.865944 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tmp-dir\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.866121 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.866088 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-home\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.866177 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.866152 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-model-cache\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.866225 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.866182 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tls-certs\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.866381 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.866369 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-dshm\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.866427 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.866392 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zz9\" (UniqueName: \"kubernetes.io/projected/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-kube-api-access-x8zz9\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.968203 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.968163 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tls-certs\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.968433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.968415 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-dshm\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.968486 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.968454 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zz9\" (UniqueName: \"kubernetes.io/projected/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-kube-api-access-x8zz9\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.968629 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.968609 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tmp-dir\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.968746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.968728 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-home\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.968810 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.968778 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-model-cache\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.969038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.969013 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tmp-dir\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.969131 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.969109 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-home\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.969194 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.969128 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-model-cache\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.971080 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.971057 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-dshm\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:27.971285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:27.971265 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tls-certs\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:28.003494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:28.003333 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zz9\" (UniqueName: \"kubernetes.io/projected/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-kube-api-access-x8zz9\") pod \"rate-test-model-33e759-kserve-97b889ddf-cfbjk\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:28.119629 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:28.112171 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:28.269798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:28.250112 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:16:28.269798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:28.250195 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:16:28.671628 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:16:28.671596 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode64f14ea_a50f_44d4_ad9b_3de00ddc3d59.slice/crio-b3db967cf9ebe6a80d4474c5d1c14efe663a2788b8644889331d69140b622b64 WatchSource:0}: Error finding container b3db967cf9ebe6a80d4474c5d1c14efe663a2788b8644889331d69140b622b64: Status 404 returned error can't find the container with id b3db967cf9ebe6a80d4474c5d1c14efe663a2788b8644889331d69140b622b64 Oct 30 01:16:28.672318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:28.672283 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk"] Oct 30 01:16:28.944376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:28.944278 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" event={"ID":"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59","Type":"ContainerStarted","Data":"55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc"} Oct 30 01:16:28.944376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:28.944329 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" event={"ID":"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59","Type":"ContainerStarted","Data":"b3db967cf9ebe6a80d4474c5d1c14efe663a2788b8644889331d69140b622b64"} Oct 30 01:16:29.005069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:29.005005 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" podStartSLOduration=2.004985412 podStartE2EDuration="2.004985412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:29.001659354 +0000 UTC m=+2527.573585579" watchObservedRunningTime="2026-10-30 01:16:29.004985412 +0000 UTC m=+2527.576911636" Oct 30 01:16:29.133132 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:29.127206 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:29.246991 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:29.246907 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:16:29.246991 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:29.246963 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:16:30.246773 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:30.246737 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:16:30.247229 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:30.246802 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:16:31.246205 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:31.246168 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:16:31.246412 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:31.246225 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 01:16:32.251127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:32.251094 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:16:32.251633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:32.251539 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:16:32.252081 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:32.252059 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:16:34.445264 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:16:34.445224 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:16:38.126891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:38.126849 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:38.140744 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:38.140713 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:39.135019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:39.134989 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:16:48.736335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:48.736293 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf"] Oct 30 01:16:53.737648 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:53.737590 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" podUID="8a137c2e-bce6-4430-9b40-751f28ed58c4" containerName="payload-processing" containerID="cri-o://324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2" gracePeriod=25 Oct 30 01:16:54.077529 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.077495 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:16:54.168444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.168412 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a137c2e-bce6-4430-9b40-751f28ed58c4-kube-api-access-wc2c4\" (UniqueName: \"kubernetes.io/projected/8a137c2e-bce6-4430-9b40-751f28ed58c4-kube-api-access-wc2c4\") pod \"8a137c2e-bce6-4430-9b40-751f28ed58c4\" (UID: \"8a137c2e-bce6-4430-9b40-751f28ed58c4\") " Oct 30 01:16:54.168666 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.168619 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a137c2e-bce6-4430-9b40-751f28ed58c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a137c2e-bce6-4430-9b40-751f28ed58c4-plugins-config-volume\") pod \"8a137c2e-bce6-4430-9b40-751f28ed58c4\" (UID: \"8a137c2e-bce6-4430-9b40-751f28ed58c4\") " Oct 30 01:16:54.169015 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.168989 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a137c2e-bce6-4430-9b40-751f28ed58c4-plugins-config-volume" pod "8a137c2e-bce6-4430-9b40-751f28ed58c4" (UID: "8a137c2e-bce6-4430-9b40-751f28ed58c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:54.169614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.169546 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a137c2e-bce6-4430-9b40-751f28ed58c4-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:16:54.170881 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.170857 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a137c2e-bce6-4430-9b40-751f28ed58c4-kube-api-access-wc2c4" pod "8a137c2e-bce6-4430-9b40-751f28ed58c4" (UID: "8a137c2e-bce6-4430-9b40-751f28ed58c4"). InnerVolumeSpecName "kube-api-access-wc2c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:54.271837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.271800 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc2c4\" (UniqueName: \"kubernetes.io/projected/8a137c2e-bce6-4430-9b40-751f28ed58c4-kube-api-access-wc2c4\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:16:54.856445 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.856404 4927 generic.go:356] "Generic (PLEG): container finished" podID="8a137c2e-bce6-4430-9b40-751f28ed58c4" containerID="324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2" exitCode=0 Oct 30 01:16:54.856947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.856476 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" Oct 30 01:16:54.856947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.856494 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" event={"ID":"8a137c2e-bce6-4430-9b40-751f28ed58c4","Type":"ContainerDied","Data":"324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2"} Oct 30 01:16:54.856947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.856556 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf" event={"ID":"8a137c2e-bce6-4430-9b40-751f28ed58c4","Type":"ContainerDied","Data":"3c79ebf8486f357bedff8c98c31cba3c412bd2458f57a21cadc9cef8d994a987"} Oct 30 01:16:54.856947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.856626 4927 scope.go:122] "RemoveContainer" containerID="324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2" Oct 30 01:16:54.872446 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.872410 4927 scope.go:122] "RemoveContainer" containerID="324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2" Oct 30 01:16:54.872743 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:16:54.872721 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2\": container with ID starting with 324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2 not found: ID does not exist" containerID="324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2" Oct 30 01:16:54.872841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.872750 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2"} err="failed to get container status \"324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2\": rpc error: code = NotFound desc = could not find container \"324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2\": container with ID starting with 324f5c417970d160881821ff15640ffb055370ef09f72151ccb67dbdc626e3d2 not found: ID does not exist" Oct 30 01:16:54.918751 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.918712 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf"] Oct 30 01:16:54.924055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:54.924027 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-65d75fbc44-75bdf"] Oct 30 01:16:55.577096 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:55.577058 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a137c2e-bce6-4430-9b40-751f28ed58c4" path="/var/lib/kubelet/pods/8a137c2e-bce6-4430-9b40-751f28ed58c4/volumes" Oct 30 01:16:57.889082 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:57.889048 4927 generic.go:356] "Generic (PLEG): container finished" podID="345e5ad3-c558-478f-a4a9-641dec5af039" containerID="a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e" exitCode=0 Oct 30 01:16:57.889556 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:57.889122 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" event={"ID":"345e5ad3-c558-478f-a4a9-641dec5af039","Type":"ContainerDied","Data":"a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e"} Oct 30 01:16:57.889695 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:57.889675 4927 scope.go:122] "RemoveContainer" containerID="a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e" Oct 30 01:16:57.891790 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:57.891750 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:16:57.891903 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:57.891832 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:16:57.891903 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:57.891872 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:16:58.911208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:58.911174 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" event={"ID":"345e5ad3-c558-478f-a4a9-641dec5af039","Type":"ContainerStarted","Data":"fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8"} Oct 30 01:16:59.710594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:59.683039 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:16:59.710594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:59.683120 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:16:59.711184 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:16:59.711132 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:17:00.599633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:00.595744 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:17:00.599633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:00.595813 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:17:01.593593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:01.593458 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:17:01.593593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:01.593527 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:17:02.598607 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:02.594784 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:17:02.599116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:02.598684 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:17:03.582917 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:03.582878 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:17:03.583226 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:03.583194 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:17:04.447250 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:17:04.444609 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:17:04.586606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:04.585938 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 01:17:04.586606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:04.586006 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 01:17:04.611557 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:04.611527 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:17:05.538092 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.538057 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls"] Oct 30 01:17:05.539083 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.539065 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8a137c2e-bce6-4430-9b40-751f28ed58c4" containerName="payload-processing" Oct 30 01:17:05.539158 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.539084 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a137c2e-bce6-4430-9b40-751f28ed58c4" containerName="payload-processing" Oct 30 01:17:05.539261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.539250 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a137c2e-bce6-4430-9b40-751f28ed58c4" containerName="payload-processing" Oct 30 01:17:05.544815 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.544788 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:05.564746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.564717 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-38b47d\"" Oct 30 01:17:05.575950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.575925 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-38b47d-dockercfg-nmfq2\"" Oct 30 01:17:05.598152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.598118 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls"] Oct 30 01:17:05.632611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.622821 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:17:05.748281 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.699021 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:17:05.760438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.760389 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4f7381-cae0-429c-97a4-c3cf138dcb35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls\" (UID: \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:05.771760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.762261 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wkx\" (UniqueName: \"kubernetes.io/projected/8e4f7381-cae0-429c-97a4-c3cf138dcb35-kube-api-access-h4wkx\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls\" (UID: \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:05.909684 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.882876 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4f7381-cae0-429c-97a4-c3cf138dcb35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls\" (UID: \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:05.909684 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.883118 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wkx\" (UniqueName: \"kubernetes.io/projected/8e4f7381-cae0-429c-97a4-c3cf138dcb35-kube-api-access-h4wkx\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls\" (UID: \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:05.909684 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:05.885611 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4f7381-cae0-429c-97a4-c3cf138dcb35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls\" (UID: \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:06.059808 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:06.059767 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wkx\" (UniqueName: \"kubernetes.io/projected/8e4f7381-cae0-429c-97a4-c3cf138dcb35-kube-api-access-h4wkx\") pod \"payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls\" (UID: \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:06.174287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:06.174198 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:06.500059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:06.500027 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls"] Oct 30 01:17:06.501165 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:17:06.501136 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4f7381_cae0_429c_97a4_c3cf138dcb35.slice/crio-1fb59992316c2530b0f1a098f8fc77963c7d2ba6433221b7b458ae9a3267b240 WatchSource:0}: Error finding container 1fb59992316c2530b0f1a098f8fc77963c7d2ba6433221b7b458ae9a3267b240: Status 404 returned error can't find the container with id 1fb59992316c2530b0f1a098f8fc77963c7d2ba6433221b7b458ae9a3267b240 Oct 30 01:17:07.346985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:07.346943 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" event={"ID":"8e4f7381-cae0-429c-97a4-c3cf138dcb35","Type":"ContainerStarted","Data":"6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f"} Oct 30 01:17:07.347474 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:07.347037 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" event={"ID":"8e4f7381-cae0-429c-97a4-c3cf138dcb35","Type":"ContainerStarted","Data":"1fb59992316c2530b0f1a098f8fc77963c7d2ba6433221b7b458ae9a3267b240"} Oct 30 01:17:07.347474 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:07.347071 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:13.825641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:13.825513 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" podStartSLOduration=8.825489778 podStartE2EDuration="8.825489778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:07.403352344 +0000 UTC m=+2565.975278569" watchObservedRunningTime="2026-10-30 01:17:13.825489778 +0000 UTC m=+2572.397416002" Oct 30 01:17:13.826200 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:13.826150 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n"] Oct 30 01:17:18.366076 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:18.366044 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:17:25.434291 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:25.434252 4927 scope.go:122] "RemoveContainer" containerID="9a3ace08c686576e00d6ae53009d414dbdc408853b33c366fe650ff6dd2993df" Oct 30 01:17:28.853120 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:28.853072 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" podUID="7524c2d1-a405-4f4b-90ff-ffd280dd1840" containerName="main" containerID="cri-o://fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714" gracePeriod=45 Oct 30 01:17:29.181003 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.180975 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:17:29.216823 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.216790 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tmp-dir\") pod \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " Oct 30 01:17:29.216997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.216885 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tls-certs\" (UniqueName: \"kubernetes.io/secret/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tls-certs\") pod \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " Oct 30 01:17:29.216997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.216905 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7524c2d1-a405-4f4b-90ff-ffd280dd1840-kube-api-access-28cq8\" (UniqueName: \"kubernetes.io/projected/7524c2d1-a405-4f4b-90ff-ffd280dd1840-kube-api-access-28cq8\") pod \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " Oct 30 01:17:29.216997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.216961 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-model-cache\") pod \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " Oct 30 01:17:29.217195 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.217010 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-dshm\") pod \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " Oct 30 01:17:29.217195 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.217064 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-home\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-home\") pod \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\" (UID: \"7524c2d1-a405-4f4b-90ff-ffd280dd1840\") " Oct 30 01:17:29.217195 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.217087 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tmp-dir" pod "7524c2d1-a405-4f4b-90ff-ffd280dd1840" (UID: "7524c2d1-a405-4f4b-90ff-ffd280dd1840"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.217383 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.217257 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-model-cache" pod "7524c2d1-a405-4f4b-90ff-ffd280dd1840" (UID: "7524c2d1-a405-4f4b-90ff-ffd280dd1840"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.217477 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.217445 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-home" pod "7524c2d1-a405-4f4b-90ff-ffd280dd1840" (UID: "7524c2d1-a405-4f4b-90ff-ffd280dd1840"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.217939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.217916 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.217939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.217939 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.218080 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.217952 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.219456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.219436 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-dshm" pod "7524c2d1-a405-4f4b-90ff-ffd280dd1840" (UID: "7524c2d1-a405-4f4b-90ff-ffd280dd1840"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.219528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.219453 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tls-certs" pod "7524c2d1-a405-4f4b-90ff-ffd280dd1840" (UID: "7524c2d1-a405-4f4b-90ff-ffd280dd1840"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:29.219528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.219457 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7524c2d1-a405-4f4b-90ff-ffd280dd1840-kube-api-access-28cq8" pod "7524c2d1-a405-4f4b-90ff-ffd280dd1840" (UID: "7524c2d1-a405-4f4b-90ff-ffd280dd1840"). InnerVolumeSpecName "kube-api-access-28cq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.320035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.320003 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7524c2d1-a405-4f4b-90ff-ffd280dd1840-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.320035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.320031 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28cq8\" (UniqueName: \"kubernetes.io/projected/7524c2d1-a405-4f4b-90ff-ffd280dd1840-kube-api-access-28cq8\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.320035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.320041 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7524c2d1-a405-4f4b-90ff-ffd280dd1840-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.673845 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.673808 4927 generic.go:356] "Generic (PLEG): container finished" podID="7524c2d1-a405-4f4b-90ff-ffd280dd1840" containerID="fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714" exitCode=0 Oct 30 01:17:29.674016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.673850 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" Oct 30 01:17:29.674016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.673872 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" event={"ID":"7524c2d1-a405-4f4b-90ff-ffd280dd1840","Type":"ContainerDied","Data":"fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714"} Oct 30 01:17:29.674016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.673911 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n" event={"ID":"7524c2d1-a405-4f4b-90ff-ffd280dd1840","Type":"ContainerDied","Data":"eaf5a829ddb57a167577db620ec31d1126554aa48704c4b08e8bb93e4b74a82c"} Oct 30 01:17:29.674016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.673930 4927 scope.go:122] "RemoveContainer" containerID="fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714" Oct 30 01:17:29.685687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.685660 4927 scope.go:122] "RemoveContainer" containerID="fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714" Oct 30 01:17:29.685953 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:17:29.685930 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714\": container with ID starting with fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714 not found: ID does not exist" containerID="fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714" Oct 30 01:17:29.685998 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.685962 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714"} err="failed to get container status \"fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714\": rpc error: code = NotFound desc = could not find container \"fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714\": container with ID starting with fd3280c96dd99404c7b601d5cd7fe50b6f087deb3065b50124b2eb7b4e367714 not found: ID does not exist" Oct 30 01:17:29.727372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.727335 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n"] Oct 30 01:17:29.737524 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:29.737498 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/sub-test-model-16c1bb-kserve-5f5496bdc5-clm5n"] Oct 30 01:17:31.575090 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:17:31.575055 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7524c2d1-a405-4f4b-90ff-ffd280dd1840" path="/var/lib/kubelet/pods/7524c2d1-a405-4f4b-90ff-ffd280dd1840/volumes" Oct 30 01:17:34.444713 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:17:34.444676 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:18:04.444754 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:18:04.444709 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:18:07.390296 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:07.390262 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f"] Oct 30 01:18:22.415257 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.415186 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" podUID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" containerName="main" containerID="cri-o://573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407" gracePeriod=45 Oct 30 01:18:22.554304 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.554257 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" podUID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": dial tcp 10.132.0.195:8000: connect: connection refused" Oct 30 01:18:22.725008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.724981 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:18:22.778660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.778624 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tls-certs\") pod \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " Oct 30 01:18:22.778844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.778696 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tmp-dir\") pod \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " Oct 30 01:18:22.778979 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.778960 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-dshm\") pod \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " Oct 30 01:18:22.780929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.778997 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-model-cache\") pod \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " Oct 30 01:18:22.780929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.779076 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-home\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-home\") pod \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " Oct 30 01:18:22.780929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.779149 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-kube-api-access-7dbz8\" (UniqueName: \"kubernetes.io/projected/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-kube-api-access-7dbz8\") pod \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\" (UID: \"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c\") " Oct 30 01:18:22.780929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.779267 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tmp-dir" pod "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" (UID: "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:22.780929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.779473 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-model-cache" pod "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" (UID: "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:22.780929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.779921 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-home" pod "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" (UID: "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:22.781882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.781857 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.782014 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.781885 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.782014 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.781900 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.790017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.783942 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-dshm" pod "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" (UID: "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:22.790017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.784016 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tls-certs" pod "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" (UID: "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:22.790017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.784079 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-kube-api-access-7dbz8" pod "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" (UID: "a8f8c2d3-40a3-4944-93f6-9b48fcc7725c"). InnerVolumeSpecName "kube-api-access-7dbz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:22.884210 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.884177 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.884210 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.884210 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dbz8\" (UniqueName: \"kubernetes.io/projected/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-kube-api-access-7dbz8\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.884407 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:22.884226 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:18:23.405509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.405467 4927 generic.go:356] "Generic (PLEG): container finished" podID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" containerID="573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407" exitCode=0 Oct 30 01:18:23.405509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.405513 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" Oct 30 01:18:23.405815 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.405554 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" event={"ID":"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c","Type":"ContainerDied","Data":"573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407"} Oct 30 01:18:23.405815 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.405633 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f" event={"ID":"a8f8c2d3-40a3-4944-93f6-9b48fcc7725c","Type":"ContainerDied","Data":"c2312781b1bb687e85fc6c1d6cf30b353276ea92b0c1168fe053fde206d2102d"} Oct 30 01:18:23.405815 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.405654 4927 scope.go:122] "RemoveContainer" containerID="573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407" Oct 30 01:18:23.419229 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.419024 4927 scope.go:122] "RemoveContainer" containerID="573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407" Oct 30 01:18:23.419514 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:18:23.419390 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407\": container with ID starting with 573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407 not found: ID does not exist" containerID="573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407" Oct 30 01:18:23.419514 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.419415 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407"} err="failed to get container status \"573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407\": rpc error: code = NotFound desc = could not find container \"573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407\": container with ID starting with 573f4be24825b158b570da8a729f9e7e4a690ad333ffa20e0e11ea6bfd5cc407 not found: ID does not exist" Oct 30 01:18:23.457479 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.457450 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f"] Oct 30 01:18:23.468962 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.468935 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f69104/auth-test-model-f69104-kserve-8565bfc4d5-f6s2f"] Oct 30 01:18:23.575978 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:18:23.575946 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" path="/var/lib/kubelet/pods/a8f8c2d3-40a3-4944-93f6-9b48fcc7725c/volumes" Oct 30 01:18:34.444438 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:18:34.444354 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:19:04.444798 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:19:04.444750 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:19:11.461763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:11.461719 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn"] Oct 30 01:19:11.465506 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:11.462630 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerName="istio-proxy" containerID="cri-o://1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445" gracePeriod=30 Oct 30 01:19:11.868579 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:11.868464 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs"] Oct 30 01:19:11.869143 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:11.869044 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" podUID="d96669c7-c61d-464d-af50-6038a619fa07" containerName="istio-proxy" containerID="cri-o://b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e" gracePeriod=30 Oct 30 01:19:16.856207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.856177 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:19:16.879920 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.879887 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-credential-socket\") pod \"152f953f-039a-4f34-a0b9-e61bf335a7c8\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " Oct 30 01:19:16.880103 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.879994 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-socket\") pod \"152f953f-039a-4f34-a0b9-e61bf335a7c8\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " Oct 30 01:19:16.880172 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.880102 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/152f953f-039a-4f34-a0b9-e61bf335a7c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152f953f-039a-4f34-a0b9-e61bf335a7c8-istiod-ca-cert\") pod \"152f953f-039a-4f34-a0b9-e61bf335a7c8\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " Oct 30 01:19:16.880287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.880164 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-credential-socket" pod "152f953f-039a-4f34-a0b9-e61bf335a7c8" (UID: "152f953f-039a-4f34-a0b9-e61bf335a7c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.880287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.880175 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-kube-api-access-4bjf8\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-kube-api-access-4bjf8\") pod \"152f953f-039a-4f34-a0b9-e61bf335a7c8\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " Oct 30 01:19:16.880287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.880222 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-podinfo\") pod \"152f953f-039a-4f34-a0b9-e61bf335a7c8\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " Oct 30 01:19:16.880455 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.880300 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-envoy\") pod \"152f953f-039a-4f34-a0b9-e61bf335a7c8\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " Oct 30 01:19:16.880455 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.880357 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-socket" pod "152f953f-039a-4f34-a0b9-e61bf335a7c8" (UID: "152f953f-039a-4f34-a0b9-e61bf335a7c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.880455 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.880405 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-token\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-token\") pod \"152f953f-039a-4f34-a0b9-e61bf335a7c8\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " Oct 30 01:19:16.880640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.880461 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-data\") pod \"152f953f-039a-4f34-a0b9-e61bf335a7c8\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " Oct 30 01:19:16.880640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.880506 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-certs\") pod \"152f953f-039a-4f34-a0b9-e61bf335a7c8\" (UID: \"152f953f-039a-4f34-a0b9-e61bf335a7c8\") " Oct 30 01:19:16.881329 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.881067 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152f953f-039a-4f34-a0b9-e61bf335a7c8-istiod-ca-cert" pod "152f953f-039a-4f34-a0b9-e61bf335a7c8" (UID: "152f953f-039a-4f34-a0b9-e61bf335a7c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:16.881485 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.881374 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152f953f-039a-4f34-a0b9-e61bf335a7c8-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:16.881485 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.881400 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:16.881485 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.881415 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:16.882013 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.881683 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-data" pod "152f953f-039a-4f34-a0b9-e61bf335a7c8" (UID: "152f953f-039a-4f34-a0b9-e61bf335a7c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.882013 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.881879 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-certs" pod "152f953f-039a-4f34-a0b9-e61bf335a7c8" (UID: "152f953f-039a-4f34-a0b9-e61bf335a7c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.883718 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.883690 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-token" pod "152f953f-039a-4f34-a0b9-e61bf335a7c8" (UID: "152f953f-039a-4f34-a0b9-e61bf335a7c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:16.890804 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.890782 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-envoy" pod "152f953f-039a-4f34-a0b9-e61bf335a7c8" (UID: "152f953f-039a-4f34-a0b9-e61bf335a7c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:16.892013 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.891984 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-podinfo" pod "152f953f-039a-4f34-a0b9-e61bf335a7c8" (UID: "152f953f-039a-4f34-a0b9-e61bf335a7c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:16.893353 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.893328 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-kube-api-access-4bjf8" pod "152f953f-039a-4f34-a0b9-e61bf335a7c8" (UID: "152f953f-039a-4f34-a0b9-e61bf335a7c8"). InnerVolumeSpecName "kube-api-access-4bjf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:16.938358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.938325 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw"] Oct 30 01:19:16.939557 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.939531 4927 cpu_manager.go:414] "containerMap: removing container" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerName="istio-proxy" Oct 30 01:19:16.939557 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.939555 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerName="istio-proxy" Oct 30 01:19:16.939798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.939599 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7524c2d1-a405-4f4b-90ff-ffd280dd1840" containerName="main" Oct 30 01:19:16.939798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.939607 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7524c2d1-a405-4f4b-90ff-ffd280dd1840" containerName="main" Oct 30 01:19:16.939798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.939622 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" containerName="main" Oct 30 01:19:16.939798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.939627 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" containerName="main" Oct 30 01:19:16.939798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.939764 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f8c2d3-40a3-4944-93f6-9b48fcc7725c" containerName="main" Oct 30 01:19:16.939798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.939784 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerName="istio-proxy" Oct 30 01:19:16.940334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.939811 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7524c2d1-a405-4f4b-90ff-ffd280dd1840" containerName="main" Oct 30 01:19:16.944679 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.944659 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:16.953405 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.953375 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw"] Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987052 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987148 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dba0780-044e-415d-995e-3c02dd91ad75-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987206 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987369 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54kkc\" (UniqueName: \"kubernetes.io/projected/3dba0780-044e-415d-995e-3c02dd91ad75-kube-api-access-54kkc\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987547 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bjf8\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-kube-api-access-4bjf8\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987566 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987597 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987612 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987626 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:16.987826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:16.987640 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152f953f-039a-4f34-a0b9-e61bf335a7c8-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.000772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.000749 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1a42e5-serving-cert\"" Oct 30 01:19:17.089124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.089090 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54kkc\" (UniqueName: \"kubernetes.io/projected/3dba0780-044e-415d-995e-3c02dd91ad75-kube-api-access-54kkc\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:17.089373 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.089353 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:17.089450 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.089425 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dba0780-044e-415d-995e-3c02dd91ad75-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:17.089513 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.089459 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:17.090820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.090745 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dba0780-044e-415d-995e-3c02dd91ad75-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:17.093844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.093818 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:17.097175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.097145 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:17.125165 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.125136 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54kkc\" (UniqueName: \"kubernetes.io/projected/3dba0780-044e-415d-995e-3c02dd91ad75-kube-api-access-54kkc\") pod \"maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:17.183073 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.181860 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:19:17.287155 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.287115 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:17.297752 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.297717 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d96669c7-c61d-464d-af50-6038a619fa07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96669c7-c61d-464d-af50-6038a619fa07-istio-podinfo\") pod \"d96669c7-c61d-464d-af50-6038a619fa07\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " Oct 30 01:19:17.297899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.297812 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-data\") pod \"d96669c7-c61d-464d-af50-6038a619fa07\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " Oct 30 01:19:17.297899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.297836 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-socket\") pod \"d96669c7-c61d-464d-af50-6038a619fa07\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " Oct 30 01:19:17.297899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.297876 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-certs\") pod \"d96669c7-c61d-464d-af50-6038a619fa07\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " Oct 30 01:19:17.298078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.297930 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-credential-socket\") pod \"d96669c7-c61d-464d-af50-6038a619fa07\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " Oct 30 01:19:17.298078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.298019 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-envoy\") pod \"d96669c7-c61d-464d-af50-6038a619fa07\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " Oct 30 01:19:17.298078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.298062 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d96669c7-c61d-464d-af50-6038a619fa07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96669c7-c61d-464d-af50-6038a619fa07-istiod-ca-cert\") pod \"d96669c7-c61d-464d-af50-6038a619fa07\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " Oct 30 01:19:17.298310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.298114 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-istio-token\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-istio-token\") pod \"d96669c7-c61d-464d-af50-6038a619fa07\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " Oct 30 01:19:17.298310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.298268 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-certs" pod "d96669c7-c61d-464d-af50-6038a619fa07" (UID: "d96669c7-c61d-464d-af50-6038a619fa07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.298425 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.298342 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-kube-api-access-ll54r\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-kube-api-access-ll54r\") pod \"d96669c7-c61d-464d-af50-6038a619fa07\" (UID: \"d96669c7-c61d-464d-af50-6038a619fa07\") " Oct 30 01:19:17.298486 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.298447 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-data" pod "d96669c7-c61d-464d-af50-6038a619fa07" (UID: "d96669c7-c61d-464d-af50-6038a619fa07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.298781 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.298758 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-socket" pod "d96669c7-c61d-464d-af50-6038a619fa07" (UID: "d96669c7-c61d-464d-af50-6038a619fa07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.299309 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.299279 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-credential-socket" pod "d96669c7-c61d-464d-af50-6038a619fa07" (UID: "d96669c7-c61d-464d-af50-6038a619fa07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.299439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.299409 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96669c7-c61d-464d-af50-6038a619fa07-istiod-ca-cert" pod "d96669c7-c61d-464d-af50-6038a619fa07" (UID: "d96669c7-c61d-464d-af50-6038a619fa07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:17.299532 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.299173 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.299655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.299640 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.299755 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.299738 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.302315 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.302291 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-kube-api-access-ll54r" pod "d96669c7-c61d-464d-af50-6038a619fa07" (UID: "d96669c7-c61d-464d-af50-6038a619fa07"). InnerVolumeSpecName "kube-api-access-ll54r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:17.302961 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.302932 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d96669c7-c61d-464d-af50-6038a619fa07-istio-podinfo" pod "d96669c7-c61d-464d-af50-6038a619fa07" (UID: "d96669c7-c61d-464d-af50-6038a619fa07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:17.303103 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.303080 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-istio-token" pod "d96669c7-c61d-464d-af50-6038a619fa07" (UID: "d96669c7-c61d-464d-af50-6038a619fa07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:17.306388 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.306359 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-envoy" pod "d96669c7-c61d-464d-af50-6038a619fa07" (UID: "d96669c7-c61d-464d-af50-6038a619fa07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:17.337019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.336921 4927 generic.go:356] "Generic (PLEG): container finished" podID="d96669c7-c61d-464d-af50-6038a619fa07" containerID="b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e" exitCode=0 Oct 30 01:19:17.337019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.336980 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" Oct 30 01:19:17.337019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.336995 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" event={"ID":"d96669c7-c61d-464d-af50-6038a619fa07","Type":"ContainerDied","Data":"b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e"} Oct 30 01:19:17.337305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.337097 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs" event={"ID":"d96669c7-c61d-464d-af50-6038a619fa07","Type":"ContainerDied","Data":"1f6cfca6c48286c0f709899f0dc8f72f74e7ad2372521c93a502a1c500a283a9"} Oct 30 01:19:17.337305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.337132 4927 scope.go:122] "RemoveContainer" containerID="b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e" Oct 30 01:19:17.340236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.340205 4927 generic.go:356] "Generic (PLEG): container finished" podID="152f953f-039a-4f34-a0b9-e61bf335a7c8" containerID="1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445" exitCode=0 Oct 30 01:19:17.340378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.340357 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" event={"ID":"152f953f-039a-4f34-a0b9-e61bf335a7c8","Type":"ContainerDied","Data":"1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445"} Oct 30 01:19:17.340441 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.340407 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" event={"ID":"152f953f-039a-4f34-a0b9-e61bf335a7c8","Type":"ContainerDied","Data":"980e362085f2a6e70618fbb104d5f9efd761905a9b7b3927cd1fc91e6ffb8e33"} Oct 30 01:19:17.340532 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.340514 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn" Oct 30 01:19:17.356129 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.356100 4927 scope.go:122] "RemoveContainer" containerID="b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e" Oct 30 01:19:17.362282 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:19:17.356551 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e\": container with ID starting with b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e not found: ID does not exist" containerID="b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e" Oct 30 01:19:17.362282 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.356651 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e"} err="failed to get container status \"b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e\": rpc error: code = NotFound desc = could not find container \"b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e\": container with ID starting with b8c69bdac3dad121cecbf7787e368cad1863dc0063641a048cb252792a8f671e not found: ID does not exist" Oct 30 01:19:17.362282 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.356676 4927 scope.go:122] "RemoveContainer" containerID="1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445" Oct 30 01:19:17.380194 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.380170 4927 scope.go:122] "RemoveContainer" containerID="1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445" Oct 30 01:19:17.381169 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:19:17.381147 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445\": container with ID starting with 1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445 not found: ID does not exist" containerID="1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445" Oct 30 01:19:17.381285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.381179 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445"} err="failed to get container status \"1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445\": rpc error: code = NotFound desc = could not find container \"1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445\": container with ID starting with 1ce1bb59cfe998ebca1d8431110a6fb9147441f9c44a8d79741fe93c51dd8445 not found: ID does not exist" Oct 30 01:19:17.402680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.402552 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.402680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.402601 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96669c7-c61d-464d-af50-6038a619fa07-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.402680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.402615 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.402680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.402629 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll54r\" (UniqueName: \"kubernetes.io/projected/d96669c7-c61d-464d-af50-6038a619fa07-kube-api-access-ll54r\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.402680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.402643 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96669c7-c61d-464d-af50-6038a619fa07-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.402680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.402657 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96669c7-c61d-464d-af50-6038a619fa07-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:17.411143 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.411112 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn"] Oct 30 01:19:17.422288 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.422248 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-n4xkn"] Oct 30 01:19:17.461111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.461078 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs"] Oct 30 01:19:17.498736 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.498695 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-57fgs"] Oct 30 01:19:17.582478 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.582436 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152f953f-039a-4f34-a0b9-e61bf335a7c8" path="/var/lib/kubelet/pods/152f953f-039a-4f34-a0b9-e61bf335a7c8/volumes" Oct 30 01:19:17.583311 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.583283 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96669c7-c61d-464d-af50-6038a619fa07" path="/var/lib/kubelet/pods/d96669c7-c61d-464d-af50-6038a619fa07/volumes" Oct 30 01:19:17.621694 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:19:17.621665 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dba0780_044e_415d_995e_3c02dd91ad75.slice/crio-f169ce6fbc9e80126e3e9633cf6fc2ec83cff1ef337f8d7db1f1e5a725530883 WatchSource:0}: Error finding container f169ce6fbc9e80126e3e9633cf6fc2ec83cff1ef337f8d7db1f1e5a725530883: Status 404 returned error can't find the container with id f169ce6fbc9e80126e3e9633cf6fc2ec83cff1ef337f8d7db1f1e5a725530883 Oct 30 01:19:17.623115 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:17.623086 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw"] Oct 30 01:19:18.355473 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:18.355374 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" event={"ID":"3dba0780-044e-415d-995e-3c02dd91ad75","Type":"ContainerStarted","Data":"c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce"} Oct 30 01:19:18.355473 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:18.355431 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" event={"ID":"3dba0780-044e-415d-995e-3c02dd91ad75","Type":"ContainerStarted","Data":"f169ce6fbc9e80126e3e9633cf6fc2ec83cff1ef337f8d7db1f1e5a725530883"} Oct 30 01:19:18.356024 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:18.355495 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:18.402867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:18.402815 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" podStartSLOduration=127.021326776 podStartE2EDuration="2m7.402800907s" totalImagesPullingTime="381.474131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:18.399019685 +0000 UTC m=+2696.970945908" watchObservedRunningTime="2026-10-30 01:19:18.402800907 +0000 UTC m=+2696.974727131" Oct 30 01:19:22.524669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:22.524635 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:19:22.528005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:22.527976 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:19:23.868545 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:23.868506 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v"] Oct 30 01:19:23.869741 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:23.869715 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d96669c7-c61d-464d-af50-6038a619fa07" containerName="istio-proxy" Oct 30 01:19:23.869741 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:23.869736 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96669c7-c61d-464d-af50-6038a619fa07" containerName="istio-proxy" Oct 30 01:19:23.869909 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:23.869878 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96669c7-c61d-464d-af50-6038a619fa07" containerName="istio-proxy" Oct 30 01:19:23.874132 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:23.874112 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:23.899158 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:23.899129 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v"] Oct 30 01:19:23.904025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:23.903987 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5f99e7-dockercfg-x7kdx\"" Oct 30 01:19:23.904302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:23.904272 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5f99e7\"" Oct 30 01:19:24.077598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:24.074670 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxf7t\" (UniqueName: \"kubernetes.io/projected/07e63105-a131-4eb5-8055-e6dcc033c00b-kube-api-access-cxf7t\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v\" (UID: \"07e63105-a131-4eb5-8055-e6dcc033c00b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:24.077598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:24.075063 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e63105-a131-4eb5-8055-e6dcc033c00b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v\" (UID: \"07e63105-a131-4eb5-8055-e6dcc033c00b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:24.241594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:24.240403 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e63105-a131-4eb5-8055-e6dcc033c00b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v\" (UID: \"07e63105-a131-4eb5-8055-e6dcc033c00b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:24.241594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:24.240534 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxf7t\" (UniqueName: \"kubernetes.io/projected/07e63105-a131-4eb5-8055-e6dcc033c00b-kube-api-access-cxf7t\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v\" (UID: \"07e63105-a131-4eb5-8055-e6dcc033c00b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:24.245593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:24.242305 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e63105-a131-4eb5-8055-e6dcc033c00b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v\" (UID: \"07e63105-a131-4eb5-8055-e6dcc033c00b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:24.292592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:24.272929 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxf7t\" (UniqueName: \"kubernetes.io/projected/07e63105-a131-4eb5-8055-e6dcc033c00b-kube-api-access-cxf7t\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v\" (UID: \"07e63105-a131-4eb5-8055-e6dcc033c00b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:24.397075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:24.397045 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:19:24.490925 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:24.490886 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:24.837096 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:19:24.837063 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e63105_a131_4eb5_8055_e6dcc033c00b.slice/crio-e8d45e11388788f587d132684817fc4c5a68cf734a1f99695e72bb1c4eccb667 WatchSource:0}: Error finding container e8d45e11388788f587d132684817fc4c5a68cf734a1f99695e72bb1c4eccb667: Status 404 returned error can't find the container with id e8d45e11388788f587d132684817fc4c5a68cf734a1f99695e72bb1c4eccb667 Oct 30 01:19:24.839998 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:24.839963 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v"] Oct 30 01:19:25.455788 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:25.455749 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" event={"ID":"07e63105-a131-4eb5-8055-e6dcc033c00b","Type":"ContainerStarted","Data":"1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8"} Oct 30 01:19:25.456166 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:25.455806 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" event={"ID":"07e63105-a131-4eb5-8055-e6dcc033c00b","Type":"ContainerStarted","Data":"e8d45e11388788f587d132684817fc4c5a68cf734a1f99695e72bb1c4eccb667"} Oct 30 01:19:25.456166 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:25.455829 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:25.498159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:25.498104 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" podStartSLOduration=2.498088703 podStartE2EDuration="2.498088703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:25.493784163 +0000 UTC m=+2704.065710384" watchObservedRunningTime="2026-10-30 01:19:25.498088703 +0000 UTC m=+2704.070014925" Oct 30 01:19:34.443816 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:19:34.443780 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:19:34.784449 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:34.784365 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d"] Oct 30 01:19:34.792094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:34.792065 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:34.809033 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:34.809008 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fc3cb2-openshift-default-dockercfg-lslw8\"" Oct 30 01:19:34.826373 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:34.826339 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d"] Oct 30 01:19:35.048259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.017968 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-envoy\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.082677 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.082628 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-podinfo\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.083018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.082997 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1038975e-5fd3-4674-9374-e4a46799b7ff-istiod-ca-cert\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.083270 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.083242 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-socket\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.083436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.083420 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-certs\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.083668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.083652 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-data\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.083899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.083886 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-token\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.084090 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.084076 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzpf\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-kube-api-access-xlzpf\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.084252 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.084239 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-credential-socket\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.229633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.223674 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-credential-socket\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.229633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.223808 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-envoy\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.257005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.256972 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-credential-socket\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.295646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.223945 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-podinfo\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.296032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.295989 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1038975e-5fd3-4674-9374-e4a46799b7ff-istiod-ca-cert\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.319514 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.297421 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1038975e-5fd3-4674-9374-e4a46799b7ff-istiod-ca-cert\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.319967 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.319930 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-socket\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.320565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.320543 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-certs\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.327785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.327755 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-data\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.336721 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.320482 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-socket\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.355661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.336964 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-token\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.355960 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.347980 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-data\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.356546 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.348414 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-certs\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.356727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.356188 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzpf\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-kube-api-access-xlzpf\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.430656 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.430144 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-token\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.477633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.467882 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzpf\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-kube-api-access-xlzpf\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.548604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.506564 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-envoy\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.548604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.519217 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-podinfo\") pod \"e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:35.709162 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:35.709126 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:36.062092 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:36.061399 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d"] Oct 30 01:19:36.068553 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:19:36.068321 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1038975e_5fd3_4674_9374_e4a46799b7ff.slice/crio-71e326bedab2894d26d53d84806d2fc0cc40056874be0b7fe9c63ba84cdd0d52 WatchSource:0}: Error finding container 71e326bedab2894d26d53d84806d2fc0cc40056874be0b7fe9c63ba84cdd0d52: Status 404 returned error can't find the container with id 71e326bedab2894d26d53d84806d2fc0cc40056874be0b7fe9c63ba84cdd0d52 Oct 30 01:19:36.072852 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:36.072810 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:19:36.072942 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:36.072915 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:19:36.072980 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:36.072946 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:19:36.469095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:36.469058 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:19:36.699076 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:36.699035 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" event={"ID":"1038975e-5fd3-4674-9374-e4a46799b7ff","Type":"ContainerStarted","Data":"35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02"} Oct 30 01:19:36.699488 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:36.699106 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" event={"ID":"1038975e-5fd3-4674-9374-e4a46799b7ff","Type":"ContainerStarted","Data":"71e326bedab2894d26d53d84806d2fc0cc40056874be0b7fe9c63ba84cdd0d52"} Oct 30 01:19:36.709255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:36.709229 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:36.744686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:36.744523 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" podStartSLOduration=2.744503872 podStartE2EDuration="2.744503872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:36.742717621 +0000 UTC m=+2715.314643846" watchObservedRunningTime="2026-10-30 01:19:36.744503872 +0000 UTC m=+2715.316430095" Oct 30 01:19:37.716879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:37.716848 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:38.721161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:38.721131 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:38.722128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:38.722104 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:19:42.990273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:42.990228 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls"] Oct 30 01:19:47.990988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:47.990941 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" podUID="8e4f7381-cae0-429c-97a4-c3cf138dcb35" containerName="payload-pre-processing" containerID="cri-o://6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f" gracePeriod=25 Oct 30 01:19:48.113178 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:48.113141 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt"] Oct 30 01:19:48.113377 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:48.113349 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" containerID="cri-o://f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17" gracePeriod=30 Oct 30 01:19:48.398070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:48.398034 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:19:48.444522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:48.444490 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4f7381-cae0-429c-97a4-c3cf138dcb35-kube-api-access-h4wkx\" (UniqueName: \"kubernetes.io/projected/8e4f7381-cae0-429c-97a4-c3cf138dcb35-kube-api-access-h4wkx\") pod \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\" (UID: \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\") " Oct 30 01:19:48.444767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:48.444717 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4f7381-cae0-429c-97a4-c3cf138dcb35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4f7381-cae0-429c-97a4-c3cf138dcb35-plugins-config-volume\") pod \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\" (UID: \"8e4f7381-cae0-429c-97a4-c3cf138dcb35\") " Oct 30 01:19:48.445334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:48.445306 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4f7381-cae0-429c-97a4-c3cf138dcb35-plugins-config-volume" pod "8e4f7381-cae0-429c-97a4-c3cf138dcb35" (UID: "8e4f7381-cae0-429c-97a4-c3cf138dcb35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:48.445492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:48.445475 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4f7381-cae0-429c-97a4-c3cf138dcb35-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:48.447989 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:48.447959 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4f7381-cae0-429c-97a4-c3cf138dcb35-kube-api-access-h4wkx" pod "8e4f7381-cae0-429c-97a4-c3cf138dcb35" (UID: "8e4f7381-cae0-429c-97a4-c3cf138dcb35"). InnerVolumeSpecName "kube-api-access-h4wkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:48.546668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:48.546634 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4wkx\" (UniqueName: \"kubernetes.io/projected/8e4f7381-cae0-429c-97a4-c3cf138dcb35-kube-api-access-h4wkx\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.032321 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.032282 4927 generic.go:356] "Generic (PLEG): container finished" podID="8e4f7381-cae0-429c-97a4-c3cf138dcb35" containerID="6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f" exitCode=0 Oct 30 01:19:49.032823 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.032340 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" Oct 30 01:19:49.032823 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.032363 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" event={"ID":"8e4f7381-cae0-429c-97a4-c3cf138dcb35","Type":"ContainerDied","Data":"6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f"} Oct 30 01:19:49.032823 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.032433 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" event={"ID":"8e4f7381-cae0-429c-97a4-c3cf138dcb35","Type":"ContainerDied","Data":"1fb59992316c2530b0f1a098f8fc77963c7d2ba6433221b7b458ae9a3267b240"} Oct 30 01:19:49.032823 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.032466 4927 scope.go:122] "RemoveContainer" containerID="6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f" Oct 30 01:19:49.055753 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.055728 4927 scope.go:122] "RemoveContainer" containerID="6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f" Oct 30 01:19:49.056082 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:19:49.056057 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f\": container with ID starting with 6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f not found: ID does not exist" containerID="6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f" Oct 30 01:19:49.056170 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.056092 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f"} err="failed to get container status \"6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f\": rpc error: code = NotFound desc = could not find container \"6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f\": container with ID starting with 6d79358c80892c2b00062a6d6b60b7a36bbd4387ae905481bff44fc15c78226f not found: ID does not exist" Oct 30 01:19:49.088944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.088912 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls"] Oct 30 01:19:49.096862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.096838 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls"] Oct 30 01:19:49.356372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.356267 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38b47d-65b5c477c9-25vls" podUID="8e4f7381-cae0-429c-97a4-c3cf138dcb35" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.203:50052\" within 1s: context deadline exceeded" Oct 30 01:19:49.356372 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:19:49.356330 4927 logging.go:55] [core] [Channel #11752 SubChannel #11753]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.203:50052", ServerName: "10.132.0.203:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.203:50052: operation was canceled" Oct 30 01:19:49.606192 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:49.606149 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4f7381-cae0-429c-97a4-c3cf138dcb35" path="/var/lib/kubelet/pods/8e4f7381-cae0-429c-97a4-c3cf138dcb35/volumes" Oct 30 01:19:53.433010 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.432978 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:19:53.530137 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530098 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-credential-socket\") pod \"060bfdb8-6cf7-403d-85d2-e81900dc442d\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " Oct 30 01:19:53.530332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530312 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-podinfo\") pod \"060bfdb8-6cf7-403d-85d2-e81900dc442d\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " Oct 30 01:19:53.530407 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530372 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-socket\") pod \"060bfdb8-6cf7-403d-85d2-e81900dc442d\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " Oct 30 01:19:53.530471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530407 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-credential-socket" pod "060bfdb8-6cf7-403d-85d2-e81900dc442d" (UID: "060bfdb8-6cf7-403d-85d2-e81900dc442d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.530521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530504 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-data\") pod \"060bfdb8-6cf7-403d-85d2-e81900dc442d\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " Oct 30 01:19:53.530584 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530540 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060bfdb8-6cf7-403d-85d2-e81900dc442d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060bfdb8-6cf7-403d-85d2-e81900dc442d-istiod-ca-cert\") pod \"060bfdb8-6cf7-403d-85d2-e81900dc442d\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " Oct 30 01:19:53.530654 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530632 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-token\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-token\") pod \"060bfdb8-6cf7-403d-85d2-e81900dc442d\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " Oct 30 01:19:53.530716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530658 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-socket" pod "060bfdb8-6cf7-403d-85d2-e81900dc442d" (UID: "060bfdb8-6cf7-403d-85d2-e81900dc442d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.530778 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530760 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-kube-api-access-2lnxd\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-kube-api-access-2lnxd\") pod \"060bfdb8-6cf7-403d-85d2-e81900dc442d\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " Oct 30 01:19:53.530879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530836 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-certs\") pod \"060bfdb8-6cf7-403d-85d2-e81900dc442d\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " Oct 30 01:19:53.530959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530883 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-envoy\") pod \"060bfdb8-6cf7-403d-85d2-e81900dc442d\" (UID: \"060bfdb8-6cf7-403d-85d2-e81900dc442d\") " Oct 30 01:19:53.531017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.530982 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060bfdb8-6cf7-403d-85d2-e81900dc442d-istiod-ca-cert" pod "060bfdb8-6cf7-403d-85d2-e81900dc442d" (UID: "060bfdb8-6cf7-403d-85d2-e81900dc442d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:53.531639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.531268 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-certs" pod "060bfdb8-6cf7-403d-85d2-e81900dc442d" (UID: "060bfdb8-6cf7-403d-85d2-e81900dc442d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.531799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.531775 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060bfdb8-6cf7-403d-85d2-e81900dc442d-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.531885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.531806 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.531885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.531821 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.531885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.531835 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.532307 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.532283 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-data" pod "060bfdb8-6cf7-403d-85d2-e81900dc442d" (UID: "060bfdb8-6cf7-403d-85d2-e81900dc442d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.533558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.533526 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-podinfo" pod "060bfdb8-6cf7-403d-85d2-e81900dc442d" (UID: "060bfdb8-6cf7-403d-85d2-e81900dc442d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:53.533701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.533559 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-token" pod "060bfdb8-6cf7-403d-85d2-e81900dc442d" (UID: "060bfdb8-6cf7-403d-85d2-e81900dc442d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:53.534643 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.534619 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-envoy" pod "060bfdb8-6cf7-403d-85d2-e81900dc442d" (UID: "060bfdb8-6cf7-403d-85d2-e81900dc442d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:53.534746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.534653 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-kube-api-access-2lnxd" pod "060bfdb8-6cf7-403d-85d2-e81900dc442d" (UID: "060bfdb8-6cf7-403d-85d2-e81900dc442d"). InnerVolumeSpecName "kube-api-access-2lnxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:53.633644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.633491 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lnxd\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-kube-api-access-2lnxd\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.633644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.633531 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.633644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.633549 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.633644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.633562 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:53.633644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:53.633626 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060bfdb8-6cf7-403d-85d2-e81900dc442d-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:19:54.080700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.080665 4927 generic.go:356] "Generic (PLEG): container finished" podID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerID="f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17" exitCode=0 Oct 30 01:19:54.080870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.080722 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" Oct 30 01:19:54.080870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.080717 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" event={"ID":"060bfdb8-6cf7-403d-85d2-e81900dc442d","Type":"ContainerDied","Data":"f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17"} Oct 30 01:19:54.080870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.080858 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt" event={"ID":"060bfdb8-6cf7-403d-85d2-e81900dc442d","Type":"ContainerDied","Data":"40423cded3c5aeacd2519cac0cc00022ca91fcf84e23e026f62eb231a58257f4"} Oct 30 01:19:54.080976 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.080890 4927 scope.go:122] "RemoveContainer" containerID="f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17" Oct 30 01:19:54.092684 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.092665 4927 scope.go:122] "RemoveContainer" containerID="53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83" Oct 30 01:19:54.103412 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.103394 4927 scope.go:122] "RemoveContainer" containerID="f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17" Oct 30 01:19:54.103716 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:19:54.103691 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17\": container with ID starting with f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17 not found: ID does not exist" containerID="f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17" Oct 30 01:19:54.103781 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.103726 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17"} err="failed to get container status \"f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17\": rpc error: code = NotFound desc = could not find container \"f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17\": container with ID starting with f5392bf90ba52a8f10b667d36ad28e45866d254321ba45d84351d38e58c6cb17 not found: ID does not exist" Oct 30 01:19:54.103781 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.103748 4927 scope.go:122] "RemoveContainer" containerID="53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83" Oct 30 01:19:54.104014 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:19:54.103995 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83\": container with ID starting with 53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83 not found: ID does not exist" containerID="53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83" Oct 30 01:19:54.104070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.104023 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83"} err="failed to get container status \"53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83\": rpc error: code = NotFound desc = could not find container \"53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83\": container with ID starting with 53d9f09819fcbdacedc52b0cad5f26cccaa64e8ee85fd7fdbf3be28a7d8bcd83 not found: ID does not exist" Oct 30 01:19:54.149341 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.149300 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt"] Oct 30 01:19:54.162203 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.162176 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ggqkt"] Oct 30 01:19:54.170309 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.170273 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m"] Oct 30 01:19:54.968867 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:54.968830 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk"] Oct 30 01:19:55.576616 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:19:55.576563 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" path="/var/lib/kubelet/pods/060bfdb8-6cf7-403d-85d2-e81900dc442d/volumes" Oct 30 01:20:04.444180 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:04.444089 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:20:09.200840 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.200795 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" podUID="008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" containerName="main" containerID="cri-o://c610d9757707a4d806493798b38e852437825f41143a3a228a8457e8ef6671a8" gracePeriod=45 Oct 30 01:20:09.409722 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.409679 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" event={"ID":"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed","Type":"ContainerDied","Data":"c610d9757707a4d806493798b38e852437825f41143a3a228a8457e8ef6671a8"} Oct 30 01:20:09.409954 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.409753 4927 generic.go:356] "Generic (PLEG): container finished" podID="008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" containerID="c610d9757707a4d806493798b38e852437825f41143a3a228a8457e8ef6671a8" exitCode=0 Oct 30 01:20:09.502069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.502039 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:20:09.585323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.585295 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-dshm\") pod \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " Oct 30 01:20:09.585488 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.585393 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tmp-dir\") pod \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " Oct 30 01:20:09.585532 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.585496 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tls-certs\") pod \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " Oct 30 01:20:09.585613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.585531 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-home\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-home\") pod \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " Oct 30 01:20:09.585672 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.585623 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-kube-api-access-p9557\" (UniqueName: \"kubernetes.io/projected/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-kube-api-access-p9557\") pod \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " Oct 30 01:20:09.585732 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.585710 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tmp-dir" pod "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" (UID: "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:09.585793 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.585753 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-model-cache\") pod \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\" (UID: \"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed\") " Oct 30 01:20:09.585912 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.585885 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-home" pod "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" (UID: "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:09.586077 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.586051 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-model-cache" pod "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" (UID: "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:09.586420 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.586401 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:09.586420 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.586421 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:09.586605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.586429 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:09.587908 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.587886 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tls-certs" pod "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" (UID: "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:09.587908 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.587900 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-kube-api-access-p9557" pod "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" (UID: "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed"). InnerVolumeSpecName "kube-api-access-p9557". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:09.588050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.588007 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-dshm" pod "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" (UID: "008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:09.687766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.687732 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9557\" (UniqueName: \"kubernetes.io/projected/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-kube-api-access-p9557\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:09.687766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.687763 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:09.687967 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:09.687778 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:10.000225 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.000181 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" podUID="e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" containerName="main" containerID="cri-o://55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc" gracePeriod=45 Oct 30 01:20:10.117519 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.117481 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" podUID="e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.202:8000/ready\": dial tcp 10.132.0.202:8000: connect: connection refused" Oct 30 01:20:10.354602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.350071 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:20:10.691926 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.691887 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-kube-api-access-x8zz9\" (UniqueName: \"kubernetes.io/projected/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-kube-api-access-x8zz9\") pod \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " Oct 30 01:20:10.692273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.692247 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tmp-dir\") pod \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " Oct 30 01:20:10.692635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.692615 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-model-cache\") pod \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " Oct 30 01:20:10.692876 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.692859 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tls-certs\") pod \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " Oct 30 01:20:10.693117 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.693099 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-dshm\") pod \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " Oct 30 01:20:10.693364 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.693347 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-home\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-home\") pod \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\" (UID: \"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59\") " Oct 30 01:20:10.701260 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.701232 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tmp-dir" pod "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" (UID: "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:10.711334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.711303 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:10.746080 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.745811 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-model-cache" pod "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" (UID: "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:10.780597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.779936 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-kube-api-access-x8zz9" pod "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" (UID: "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59"). InnerVolumeSpecName "kube-api-access-x8zz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:10.819552 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.819503 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4"] Oct 30 01:20:10.821099 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821070 4927 cpu_manager.go:414] "containerMap: removing container" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" Oct 30 01:20:10.821099 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821093 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" Oct 30 01:20:10.821324 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821121 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8e4f7381-cae0-429c-97a4-c3cf138dcb35" containerName="payload-pre-processing" Oct 30 01:20:10.821324 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821127 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4f7381-cae0-429c-97a4-c3cf138dcb35" containerName="payload-pre-processing" Oct 30 01:20:10.821324 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821196 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" containerName="main" Oct 30 01:20:10.821324 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821201 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" containerName="main" Oct 30 01:20:10.821324 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821209 4927 cpu_manager.go:414] "containerMap: removing container" podUID="008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" containerName="main" Oct 30 01:20:10.821324 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821215 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" containerName="main" Oct 30 01:20:10.821707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821491 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" Oct 30 01:20:10.821707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821507 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" Oct 30 01:20:10.821707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821515 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4f7381-cae0-429c-97a4-c3cf138dcb35" containerName="payload-pre-processing" Oct 30 01:20:10.821707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821536 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" containerName="main" Oct 30 01:20:10.821707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821541 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" containerName="main" Oct 30 01:20:10.821960 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.821701 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-home" pod "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" (UID: "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:10.831957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.831933 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:10.832152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.832131 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zz9\" (UniqueName: \"kubernetes.io/projected/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-kube-api-access-x8zz9\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:10.832264 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.832250 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:10.833271 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.833248 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-dshm" pod "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" (UID: "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:10.833479 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.833462 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:10.834363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.834343 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tls-certs" pod "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" (UID: "e64f14ea-a50f-44d4-ad9b-3de00ddc3d59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:10.834491 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.834395 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4"] Oct 30 01:20:10.848642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.848605 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" event={"ID":"008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed","Type":"ContainerDied","Data":"7ad7309911d99afa74d41354f9ed5fcb19d32cb2f359657acc551ba45d49db53"} Oct 30 01:20:10.848810 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.848614 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m" Oct 30 01:20:10.849675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.849610 4927 scope.go:122] "RemoveContainer" containerID="c610d9757707a4d806493798b38e852437825f41143a3a228a8457e8ef6671a8" Oct 30 01:20:10.851101 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.851086 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5f99e7-serving-cert\"" Oct 30 01:20:10.852480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.852450 4927 generic.go:356] "Generic (PLEG): container finished" podID="e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" containerID="55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc" exitCode=0 Oct 30 01:20:10.852683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.852664 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" event={"ID":"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59","Type":"ContainerDied","Data":"55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc"} Oct 30 01:20:10.852782 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.852699 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" event={"ID":"e64f14ea-a50f-44d4-ad9b-3de00ddc3d59","Type":"ContainerDied","Data":"b3db967cf9ebe6a80d4474c5d1c14efe663a2788b8644889331d69140b622b64"} Oct 30 01:20:10.852826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.852791 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk" Oct 30 01:20:10.863679 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.863657 4927 scope.go:122] "RemoveContainer" containerID="55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc" Oct 30 01:20:10.878873 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.878854 4927 scope.go:122] "RemoveContainer" containerID="55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc" Oct 30 01:20:10.879147 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:10.879128 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc\": container with ID starting with 55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc not found: ID does not exist" containerID="55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc" Oct 30 01:20:10.879238 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.879155 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc"} err="failed to get container status \"55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc\": rpc error: code = NotFound desc = could not find container \"55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc\": container with ID starting with 55ce6a11263221672b7a8421e1d7415e2836a957aafd6abe195ef9c0cb46dddc not found: ID does not exist" Oct 30 01:20:10.913517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.913487 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m"] Oct 30 01:20:10.919751 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.919728 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c4f6f/rate-test-model-6c4f6f-kserve-86c544b44f-rpw4m"] Oct 30 01:20:10.934212 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.934188 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vhjw\" (UniqueName: \"kubernetes.io/projected/29ab4ce5-416f-4f47-832a-da72d1263042-kube-api-access-7vhjw\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:10.934323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.934305 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:10.934380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.934338 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-metrics-certs\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:10.934608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.934587 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab4ce5-416f-4f47-832a-da72d1263042-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:10.934750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.934733 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:10.934801 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.934755 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:10.954251 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.954185 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk"] Oct 30 01:20:10.958972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:10.958950 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-33e759/rate-test-model-33e759-kserve-97b889ddf-cfbjk"] Oct 30 01:20:11.036216 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.036183 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:11.036374 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.036232 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-metrics-certs\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:11.036454 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.036419 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab4ce5-416f-4f47-832a-da72d1263042-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:11.036600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.036567 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vhjw\" (UniqueName: \"kubernetes.io/projected/29ab4ce5-416f-4f47-832a-da72d1263042-kube-api-access-7vhjw\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:11.037135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.037109 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab4ce5-416f-4f47-832a-da72d1263042-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:11.038954 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.038923 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-metrics-certs\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:11.039047 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.038975 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:11.059383 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.059355 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vhjw\" (UniqueName: \"kubernetes.io/projected/29ab4ce5-416f-4f47-832a-da72d1263042-kube-api-access-7vhjw\") pod \"maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:11.177873 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.177836 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:11.503744 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:20:11.503711 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ab4ce5_416f_4f47_832a_da72d1263042.slice/crio-e8d876cddef7a022365252366930373f24c472ad0e9153586cd45ac38ff3c869 WatchSource:0}: Error finding container e8d876cddef7a022365252366930373f24c472ad0e9153586cd45ac38ff3c869: Status 404 returned error can't find the container with id e8d876cddef7a022365252366930373f24c472ad0e9153586cd45ac38ff3c869 Oct 30 01:20:11.504175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.504151 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4"] Oct 30 01:20:11.505192 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.505171 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:20:11.602837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.602804 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed" path="/var/lib/kubelet/pods/008ecceb-3fa1-4cb8-8c30-74eafaa4f7ed/volumes" Oct 30 01:20:11.603457 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.603436 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64f14ea-a50f-44d4-ad9b-3de00ddc3d59" path="/var/lib/kubelet/pods/e64f14ea-a50f-44d4-ad9b-3de00ddc3d59/volumes" Oct 30 01:20:11.865688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:11.865589 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" event={"ID":"29ab4ce5-416f-4f47-832a-da72d1263042","Type":"ContainerStarted","Data":"e8d876cddef7a022365252366930373f24c472ad0e9153586cd45ac38ff3c869"} Oct 30 01:20:12.878355 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:12.878318 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" event={"ID":"29ab4ce5-416f-4f47-832a-da72d1263042","Type":"ContainerStarted","Data":"06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865"} Oct 30 01:20:12.878824 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:12.878430 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:12.924984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:12.924930 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" podStartSLOduration=2.469328627 podStartE2EDuration="2.924915354s" totalImagesPullingTime="455.586727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:12.921798739 +0000 UTC m=+2751.493724972" watchObservedRunningTime="2026-10-30 01:20:12.924915354 +0000 UTC m=+2751.496841575" Oct 30 01:20:14.057602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.057555 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc"] Oct 30 01:20:14.058124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.057809 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" podUID="74f52ae8-27a3-46a3-9bd0-509b6397a318" containerName="maas-api" containerID="cri-o://ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25" gracePeriod=30 Oct 30 01:20:14.153604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.153550 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" podUID="74f52ae8-27a3-46a3-9bd0-509b6397a318" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.187:8443/health\": dial tcp 10.132.0.187:8443: connect: connection refused" Oct 30 01:20:14.216319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.216279 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8"] Oct 30 01:20:14.216517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.216494 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" podUID="636f26b2-b3ba-46d6-95fb-b49ad3b9315e" containerName="payload-processing" containerID="cri-o://7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e" gracePeriod=30 Oct 30 01:20:14.254108 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.252675 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w"] Oct 30 01:20:14.254108 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.252934 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" podUID="6d2cff76-9eb0-4def-aba6-0a033eaecd42" containerName="payload-pre-processing" containerID="cri-o://46ea7c73452e558247db57c4748c5e1c20673808dda7de9c26c7e96dc64f333b" gracePeriod=30 Oct 30 01:20:14.411428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.411399 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:20:14.514300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.514258 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-metrics-certs\") pod \"74f52ae8-27a3-46a3-9bd0-509b6397a318\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " Oct 30 01:20:14.514506 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.514326 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls\") pod \"74f52ae8-27a3-46a3-9bd0-509b6397a318\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " Oct 30 01:20:14.514506 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.514370 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f52ae8-27a3-46a3-9bd0-509b6397a318-kube-api-access-q678t\" (UniqueName: \"kubernetes.io/projected/74f52ae8-27a3-46a3-9bd0-509b6397a318-kube-api-access-q678t\") pod \"74f52ae8-27a3-46a3-9bd0-509b6397a318\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " Oct 30 01:20:14.514704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.514680 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f52ae8-27a3-46a3-9bd0-509b6397a318-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52ae8-27a3-46a3-9bd0-509b6397a318-openshift-service-ca\") pod \"74f52ae8-27a3-46a3-9bd0-509b6397a318\" (UID: \"74f52ae8-27a3-46a3-9bd0-509b6397a318\") " Oct 30 01:20:14.515923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.515894 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f52ae8-27a3-46a3-9bd0-509b6397a318-openshift-service-ca" pod "74f52ae8-27a3-46a3-9bd0-509b6397a318" (UID: "74f52ae8-27a3-46a3-9bd0-509b6397a318"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:14.518936 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.518908 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-metrics-certs" pod "74f52ae8-27a3-46a3-9bd0-509b6397a318" (UID: "74f52ae8-27a3-46a3-9bd0-509b6397a318"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:14.519685 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.519657 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls" pod "74f52ae8-27a3-46a3-9bd0-509b6397a318" (UID: "74f52ae8-27a3-46a3-9bd0-509b6397a318"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:14.521389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.521358 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f52ae8-27a3-46a3-9bd0-509b6397a318-kube-api-access-q678t" pod "74f52ae8-27a3-46a3-9bd0-509b6397a318" (UID: "74f52ae8-27a3-46a3-9bd0-509b6397a318"). InnerVolumeSpecName "kube-api-access-q678t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:14.556018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.555648 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:20:14.650594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.635661 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:14.650594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.635705 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52ae8-27a3-46a3-9bd0-509b6397a318-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:14.650594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.635720 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q678t\" (UniqueName: \"kubernetes.io/projected/74f52ae8-27a3-46a3-9bd0-509b6397a318-kube-api-access-q678t\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:14.650594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.635736 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52ae8-27a3-46a3-9bd0-509b6397a318-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:14.765597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.759378 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-kube-api-access-qkszq\" (UniqueName: \"kubernetes.io/projected/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-kube-api-access-qkszq\") pod \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\" (UID: \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\") " Oct 30 01:20:14.765597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.759656 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-plugins-config-volume\") pod \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\" (UID: \"636f26b2-b3ba-46d6-95fb-b49ad3b9315e\") " Oct 30 01:20:14.810883 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.810839 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-plugins-config-volume" pod "636f26b2-b3ba-46d6-95fb-b49ad3b9315e" (UID: "636f26b2-b3ba-46d6-95fb-b49ad3b9315e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:14.851198 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.850875 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-kube-api-access-qkszq" pod "636f26b2-b3ba-46d6-95fb-b49ad3b9315e" (UID: "636f26b2-b3ba-46d6-95fb-b49ad3b9315e"). InnerVolumeSpecName "kube-api-access-qkszq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:14.877903 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.877865 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:14.877903 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.877906 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkszq\" (UniqueName: \"kubernetes.io/projected/636f26b2-b3ba-46d6-95fb-b49ad3b9315e-kube-api-access-qkszq\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:14.911123 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.911084 4927 generic.go:356] "Generic (PLEG): container finished" podID="74f52ae8-27a3-46a3-9bd0-509b6397a318" containerID="ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25" exitCode=0 Oct 30 01:20:14.911300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.911245 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" event={"ID":"74f52ae8-27a3-46a3-9bd0-509b6397a318","Type":"ContainerDied","Data":"ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25"} Oct 30 01:20:14.911377 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.911309 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" event={"ID":"74f52ae8-27a3-46a3-9bd0-509b6397a318","Type":"ContainerDied","Data":"db1b28f662f1220f0fc95183bd99f06fa228a081c08a2fcb8d33e400823aa179"} Oct 30 01:20:14.911377 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.911340 4927 scope.go:122] "RemoveContainer" containerID="ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25" Oct 30 01:20:14.911512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.911495 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc" Oct 30 01:20:14.921327 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.921296 4927 generic.go:356] "Generic (PLEG): container finished" podID="636f26b2-b3ba-46d6-95fb-b49ad3b9315e" containerID="7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e" exitCode=0 Oct 30 01:20:14.921617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.921554 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" event={"ID":"636f26b2-b3ba-46d6-95fb-b49ad3b9315e","Type":"ContainerDied","Data":"7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e"} Oct 30 01:20:14.921708 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.921646 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" event={"ID":"636f26b2-b3ba-46d6-95fb-b49ad3b9315e","Type":"ContainerDied","Data":"7a40ac70e578c8fa6701b8640d6a3cd45ff07741329edaa30bd838be5e6de6b5"} Oct 30 01:20:14.921764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.921723 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8" Oct 30 01:20:14.926494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.926464 4927 generic.go:356] "Generic (PLEG): container finished" podID="6d2cff76-9eb0-4def-aba6-0a033eaecd42" containerID="46ea7c73452e558247db57c4748c5e1c20673808dda7de9c26c7e96dc64f333b" exitCode=0 Oct 30 01:20:14.926642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.926597 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:20:14.926642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.926616 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" event={"ID":"6d2cff76-9eb0-4def-aba6-0a033eaecd42","Type":"ContainerDied","Data":"46ea7c73452e558247db57c4748c5e1c20673808dda7de9c26c7e96dc64f333b"} Oct 30 01:20:14.935963 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.935940 4927 scope.go:122] "RemoveContainer" containerID="ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25" Oct 30 01:20:14.936412 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:14.936384 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25\": container with ID starting with ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25 not found: ID does not exist" containerID="ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25" Oct 30 01:20:14.936520 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.936423 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25"} err="failed to get container status \"ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25\": rpc error: code = NotFound desc = could not find container \"ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25\": container with ID starting with ba16567fe81fb1ba72995d3336cb2e19543f863f9d5117c4d544abeb8ecd5d25 not found: ID does not exist" Oct 30 01:20:14.936520 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.936447 4927 scope.go:122] "RemoveContainer" containerID="7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e" Oct 30 01:20:14.949774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.949751 4927 scope.go:122] "RemoveContainer" containerID="7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e" Oct 30 01:20:14.950071 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:14.950053 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e\": container with ID starting with 7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e not found: ID does not exist" containerID="7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e" Oct 30 01:20:14.950135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.950083 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e"} err="failed to get container status \"7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e\": rpc error: code = NotFound desc = could not find container \"7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e\": container with ID starting with 7977dee609f75df514732460b68cd097d92144c688e16946d1dc5939fc0aba7e not found: ID does not exist" Oct 30 01:20:14.950135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.950098 4927 scope.go:122] "RemoveContainer" containerID="46ea7c73452e558247db57c4748c5e1c20673808dda7de9c26c7e96dc64f333b" Oct 30 01:20:14.982911 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.982882 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d2cff76-9eb0-4def-aba6-0a033eaecd42-kube-api-access-jrw7z\" (UniqueName: \"kubernetes.io/projected/6d2cff76-9eb0-4def-aba6-0a033eaecd42-kube-api-access-jrw7z\") pod \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\" (UID: \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\") " Oct 30 01:20:14.983053 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.983004 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d2cff76-9eb0-4def-aba6-0a033eaecd42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2cff76-9eb0-4def-aba6-0a033eaecd42-plugins-config-volume\") pod \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\" (UID: \"6d2cff76-9eb0-4def-aba6-0a033eaecd42\") " Oct 30 01:20:14.983358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.983332 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2cff76-9eb0-4def-aba6-0a033eaecd42-plugins-config-volume" pod "6d2cff76-9eb0-4def-aba6-0a033eaecd42" (UID: "6d2cff76-9eb0-4def-aba6-0a033eaecd42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:14.983990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.983966 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2cff76-9eb0-4def-aba6-0a033eaecd42-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:14.985155 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:14.985131 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2cff76-9eb0-4def-aba6-0a033eaecd42-kube-api-access-jrw7z" pod "6d2cff76-9eb0-4def-aba6-0a033eaecd42" (UID: "6d2cff76-9eb0-4def-aba6-0a033eaecd42"). InnerVolumeSpecName "kube-api-access-jrw7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:15.025738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.025706 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc"] Oct 30 01:20:15.038993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.038964 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91e87-7db74cf9f5-9zjzc"] Oct 30 01:20:15.108037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.106445 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrw7z\" (UniqueName: \"kubernetes.io/projected/6d2cff76-9eb0-4def-aba6-0a033eaecd42-kube-api-access-jrw7z\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:15.124707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.124630 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8"] Oct 30 01:20:15.149347 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.148333 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-7bf7b54d6-7nvt8"] Oct 30 01:20:15.575792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.575754 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636f26b2-b3ba-46d6-95fb-b49ad3b9315e" path="/var/lib/kubelet/pods/636f26b2-b3ba-46d6-95fb-b49ad3b9315e/volumes" Oct 30 01:20:15.576443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.576418 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f52ae8-27a3-46a3-9bd0-509b6397a318" path="/var/lib/kubelet/pods/74f52ae8-27a3-46a3-9bd0-509b6397a318/volumes" Oct 30 01:20:15.936520 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.936484 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" Oct 30 01:20:15.936520 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.936511 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w" event={"ID":"6d2cff76-9eb0-4def-aba6-0a033eaecd42","Type":"ContainerDied","Data":"75453d7eeab2369698237690a844b6409bb461eebae0438ac0f0b320a7abfaea"} Oct 30 01:20:15.988613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.988582 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w"] Oct 30 01:20:15.999251 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:15.999218 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f91e87-7c488fc6f9-bkf7w"] Oct 30 01:20:17.649627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:17.642809 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d2cff76-9eb0-4def-aba6-0a033eaecd42" path="/var/lib/kubelet/pods/6d2cff76-9eb0-4def-aba6-0a033eaecd42/volumes" Oct 30 01:20:18.605436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:18.604695 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:20:18.940125 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:18.939706 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:20:21.996490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:21.996446 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z"] Oct 30 01:20:21.996951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:21.996695 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" containerID="cri-o://9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff" gracePeriod=30 Oct 30 01:20:27.064773 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:27.064737 4927 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c66b95_acdf_4fbf_997f_01e779dac926.slice/crio-conmon-9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:20:27.354200 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.354168 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:20:27.422304 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422261 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-credential-socket\") pod \"05c66b95-acdf-4fbf-997f-01e779dac926\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " Oct 30 01:20:27.422470 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422319 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-kube-api-access-ttv9t\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-kube-api-access-ttv9t\") pod \"05c66b95-acdf-4fbf-997f-01e779dac926\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " Oct 30 01:20:27.422470 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422423 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-socket\") pod \"05c66b95-acdf-4fbf-997f-01e779dac926\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " Oct 30 01:20:27.422628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422472 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-certs\") pod \"05c66b95-acdf-4fbf-997f-01e779dac926\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " Oct 30 01:20:27.422628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422528 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05c66b95-acdf-4fbf-997f-01e779dac926-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c66b95-acdf-4fbf-997f-01e779dac926-istio-podinfo\") pod \"05c66b95-acdf-4fbf-997f-01e779dac926\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " Oct 30 01:20:27.422628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422610 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-credential-socket" pod "05c66b95-acdf-4fbf-997f-01e779dac926" (UID: "05c66b95-acdf-4fbf-997f-01e779dac926"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:27.422822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422699 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-istio-token\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-istio-token\") pod \"05c66b95-acdf-4fbf-997f-01e779dac926\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " Oct 30 01:20:27.422822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422781 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c66b95-acdf-4fbf-997f-01e779dac926-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c66b95-acdf-4fbf-997f-01e779dac926-istiod-ca-cert\") pod \"05c66b95-acdf-4fbf-997f-01e779dac926\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " Oct 30 01:20:27.422822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422790 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-socket" pod "05c66b95-acdf-4fbf-997f-01e779dac926" (UID: "05c66b95-acdf-4fbf-997f-01e779dac926"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:27.422988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422836 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-envoy\") pod \"05c66b95-acdf-4fbf-997f-01e779dac926\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " Oct 30 01:20:27.422988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422889 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-certs" pod "05c66b95-acdf-4fbf-997f-01e779dac926" (UID: "05c66b95-acdf-4fbf-997f-01e779dac926"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:27.422988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.422953 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-data\") pod \"05c66b95-acdf-4fbf-997f-01e779dac926\" (UID: \"05c66b95-acdf-4fbf-997f-01e779dac926\") " Oct 30 01:20:27.423219 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.423192 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c66b95-acdf-4fbf-997f-01e779dac926-istiod-ca-cert" pod "05c66b95-acdf-4fbf-997f-01e779dac926" (UID: "05c66b95-acdf-4fbf-997f-01e779dac926"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:27.423322 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.423298 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-data" pod "05c66b95-acdf-4fbf-997f-01e779dac926" (UID: "05c66b95-acdf-4fbf-997f-01e779dac926"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:27.423953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.423929 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c66b95-acdf-4fbf-997f-01e779dac926-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.424045 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.423956 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.424045 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.423969 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.424045 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.423984 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.424045 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.423997 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.424982 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.424951 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-istio-token" pod "05c66b95-acdf-4fbf-997f-01e779dac926" (UID: "05c66b95-acdf-4fbf-997f-01e779dac926"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:27.425319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.425291 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-kube-api-access-ttv9t" pod "05c66b95-acdf-4fbf-997f-01e779dac926" (UID: "05c66b95-acdf-4fbf-997f-01e779dac926"). InnerVolumeSpecName "kube-api-access-ttv9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:27.425434 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.425325 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-envoy" pod "05c66b95-acdf-4fbf-997f-01e779dac926" (UID: "05c66b95-acdf-4fbf-997f-01e779dac926"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:27.425972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.425948 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05c66b95-acdf-4fbf-997f-01e779dac926-istio-podinfo" pod "05c66b95-acdf-4fbf-997f-01e779dac926" (UID: "05c66b95-acdf-4fbf-997f-01e779dac926"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:20:27.496950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.496916 4927 generic.go:356] "Generic (PLEG): container finished" podID="05c66b95-acdf-4fbf-997f-01e779dac926" containerID="9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff" exitCode=0 Oct 30 01:20:27.497124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.496970 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" Oct 30 01:20:27.497124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.496997 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" event={"ID":"05c66b95-acdf-4fbf-997f-01e779dac926","Type":"ContainerDied","Data":"9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff"} Oct 30 01:20:27.497124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.497054 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z" event={"ID":"05c66b95-acdf-4fbf-997f-01e779dac926","Type":"ContainerDied","Data":"3686e58780aaddd5442ac0279b4c700207154295b329efaa68b63ba9a4908f43"} Oct 30 01:20:27.497124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.497078 4927 scope.go:122] "RemoveContainer" containerID="9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff" Oct 30 01:20:27.510235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.510218 4927 scope.go:122] "RemoveContainer" containerID="9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff" Oct 30 01:20:27.510499 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:27.510480 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff\": container with ID starting with 9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff not found: ID does not exist" containerID="9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff" Oct 30 01:20:27.510553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.510507 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff"} err="failed to get container status \"9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff\": rpc error: code = NotFound desc = could not find container \"9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff\": container with ID starting with 9e00364ad18ceb094069066bc2d85e34b613df6478edf6f59493c6c70a7c6bff not found: ID does not exist" Oct 30 01:20:27.526046 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.526017 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c66b95-acdf-4fbf-997f-01e779dac926-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.526046 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.526043 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.526190 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.526053 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c66b95-acdf-4fbf-997f-01e779dac926-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.526190 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.526063 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttv9t\" (UniqueName: \"kubernetes.io/projected/05c66b95-acdf-4fbf-997f-01e779dac926-kube-api-access-ttv9t\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:27.550122 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.550095 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z"] Oct 30 01:20:27.554817 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.554793 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-std5z"] Oct 30 01:20:27.573769 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:27.573709 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" path="/var/lib/kubelet/pods/05c66b95-acdf-4fbf-997f-01e779dac926/volumes" Oct 30 01:20:29.316646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:29.316611 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92"] Oct 30 01:20:29.358071 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:29.358034 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb"] Oct 30 01:20:30.017597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.016725 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4"] Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018062 4927 cpu_manager.go:414] "containerMap: removing container" podUID="74f52ae8-27a3-46a3-9bd0-509b6397a318" containerName="maas-api" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018089 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f52ae8-27a3-46a3-9bd0-509b6397a318" containerName="maas-api" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018138 4927 cpu_manager.go:414] "containerMap: removing container" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018147 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018183 4927 cpu_manager.go:414] "containerMap: removing container" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018192 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060bfdb8-6cf7-403d-85d2-e81900dc442d" containerName="istio-proxy" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018236 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6d2cff76-9eb0-4def-aba6-0a033eaecd42" containerName="payload-pre-processing" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018244 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d2cff76-9eb0-4def-aba6-0a033eaecd42" containerName="payload-pre-processing" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018265 4927 cpu_manager.go:414] "containerMap: removing container" podUID="636f26b2-b3ba-46d6-95fb-b49ad3b9315e" containerName="payload-processing" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018274 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636f26b2-b3ba-46d6-95fb-b49ad3b9315e" containerName="payload-processing" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018459 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d2cff76-9eb0-4def-aba6-0a033eaecd42" containerName="payload-pre-processing" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018477 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="636f26b2-b3ba-46d6-95fb-b49ad3b9315e" containerName="payload-processing" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018491 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f52ae8-27a3-46a3-9bd0-509b6397a318" containerName="maas-api" Oct 30 01:20:30.025593 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.018503 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c66b95-acdf-4fbf-997f-01e779dac926" containerName="istio-proxy" Oct 30 01:20:30.051924 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.051891 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.095595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.074196 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4"] Oct 30 01:20:30.096424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.082457 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6cba49-serving-cert\"" Oct 30 01:20:30.155597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.142064 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-metrics-certs\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.155597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.142131 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcvz\" (UniqueName: \"kubernetes.io/projected/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-kube-api-access-6wcvz\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.155597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.142254 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.155597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.142395 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.244355 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.244321 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-metrics-certs\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.244528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.244370 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcvz\" (UniqueName: \"kubernetes.io/projected/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-kube-api-access-6wcvz\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.244528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.244459 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.244634 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.244550 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.244683 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:30.244632 4927 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-serving-cert: secret "maas-api-e2e-ipp-b-6cba49-serving-cert" not found Oct 30 01:20:30.244726 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:30.244704 4927 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls podName:1cfdc6a3-6c99-405e-a071-1ee80c4e6911 nodeName:}" failed. No retries permitted until 2026-10-30 01:20:30.744683791 +0000 UTC m=+2769.316610004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls") pod "maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" (UID: "1cfdc6a3-6c99-405e-a071-1ee80c4e6911") : secret "maas-api-e2e-ipp-b-6cba49-serving-cert" not found Oct 30 01:20:30.245214 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.245190 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.246894 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.246872 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-metrics-certs\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.266952 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.266931 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcvz\" (UniqueName: \"kubernetes.io/projected/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-kube-api-access-6wcvz\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.760792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.760757 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:30.763301 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:30.763272 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:31.020137 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:31.020048 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:31.339891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:31.339858 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4"] Oct 30 01:20:31.340526 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:20:31.340495 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfdc6a3_6c99_405e_a071_1ee80c4e6911.slice/crio-d1fb5d4fc3f7979ec3e1385f009c6f1560d9bee5cdc60a5514cd24d8ed14efd3 WatchSource:0}: Error finding container d1fb5d4fc3f7979ec3e1385f009c6f1560d9bee5cdc60a5514cd24d8ed14efd3: Status 404 returned error can't find the container with id d1fb5d4fc3f7979ec3e1385f009c6f1560d9bee5cdc60a5514cd24d8ed14efd3 Oct 30 01:20:31.537804 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:31.537767 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" event={"ID":"1cfdc6a3-6c99-405e-a071-1ee80c4e6911","Type":"ContainerStarted","Data":"d1fb5d4fc3f7979ec3e1385f009c6f1560d9bee5cdc60a5514cd24d8ed14efd3"} Oct 30 01:20:32.086770 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.086688 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx"] Oct 30 01:20:32.091869 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.091846 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.114067 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.114025 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-fc3cb2-serving-cert\"" Oct 30 01:20:32.122302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.122271 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx"] Oct 30 01:20:32.199862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.199815 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80835022-7789-49ed-b568-c8f28ae3ae2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.200065 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.199933 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-metrics-certs\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.200226 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.200206 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cks46\" (UniqueName: \"kubernetes.io/projected/80835022-7789-49ed-b568-c8f28ae3ae2c-kube-api-access-cks46\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.200294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.200241 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.309944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.309904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cks46\" (UniqueName: \"kubernetes.io/projected/80835022-7789-49ed-b568-c8f28ae3ae2c-kube-api-access-cks46\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.310165 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.309970 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.310165 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.310065 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80835022-7789-49ed-b568-c8f28ae3ae2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.310165 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.310125 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-metrics-certs\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.313466 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.312052 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80835022-7789-49ed-b568-c8f28ae3ae2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.351460 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.351234 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-metrics-certs\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.351460 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.351386 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cks46\" (UniqueName: \"kubernetes.io/projected/80835022-7789-49ed-b568-c8f28ae3ae2c-kube-api-access-cks46\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.356161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.355150 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.409269 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.409233 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:32.561459 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.561420 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" event={"ID":"1cfdc6a3-6c99-405e-a071-1ee80c4e6911","Type":"ContainerStarted","Data":"da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6"} Oct 30 01:20:32.561676 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.561508 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:32.604273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.603886 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" podStartSLOduration=3.145206287 podStartE2EDuration="3.603863956s" totalImagesPullingTime="458.657669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:32.602838998 +0000 UTC m=+2771.174765219" watchObservedRunningTime="2026-10-30 01:20:32.603863956 +0000 UTC m=+2771.175790181" Oct 30 01:20:32.757003 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:32.756955 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx"] Oct 30 01:20:33.694879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:33.694827 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" event={"ID":"80835022-7789-49ed-b568-c8f28ae3ae2c","Type":"ContainerStarted","Data":"fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7"} Oct 30 01:20:33.695414 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:33.695249 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" event={"ID":"80835022-7789-49ed-b568-c8f28ae3ae2c","Type":"ContainerStarted","Data":"b89d2b41397bb396b1a237e09b603aabc9190dea5df6a6b5c655b8a631a181bf"} Oct 30 01:20:33.695414 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:33.695314 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:33.735554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:33.735495 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" podStartSLOduration=1.42549597 podStartE2EDuration="1.735480813s" totalImagesPullingTime="309.984843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:33.734710848 +0000 UTC m=+2772.306637072" watchObservedRunningTime="2026-10-30 01:20:33.735480813 +0000 UTC m=+2772.307407034" Oct 30 01:20:34.317468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:34.317391 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" podUID="2677740b-3783-4ff1-8ff8-7c05c34a7ab5" containerName="payload-processing" containerID="cri-o://2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85" gracePeriod=25 Oct 30 01:20:34.359043 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:34.359006 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" podUID="538ea5e7-d6f3-4391-97f6-4f7389bddb83" containerName="payload-pre-processing" containerID="cri-o://6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087" gracePeriod=25 Oct 30 01:20:34.444440 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:34.444398 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:20:37.399462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:37.399420 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" podUID="538ea5e7-d6f3-4391-97f6-4f7389bddb83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:20:38.404739 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:38.404683 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" podUID="2677740b-3783-4ff1-8ff8-7c05c34a7ab5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:20:38.704287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:38.704208 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:20:39.715152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:39.715118 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:20:41.080764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.080727 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw"] Oct 30 01:20:41.081195 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.080930 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" podUID="3dba0780-044e-415d-995e-3c02dd91ad75" containerName="maas-api" containerID="cri-o://c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce" gracePeriod=30 Oct 30 01:20:41.775132 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.775100 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:20:41.838156 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.838053 4927 generic.go:356] "Generic (PLEG): container finished" podID="3dba0780-044e-415d-995e-3c02dd91ad75" containerID="c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce" exitCode=0 Oct 30 01:20:41.838156 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.838103 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" event={"ID":"3dba0780-044e-415d-995e-3c02dd91ad75","Type":"ContainerDied","Data":"c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce"} Oct 30 01:20:41.838156 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.838107 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" Oct 30 01:20:41.838156 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.838151 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw" event={"ID":"3dba0780-044e-415d-995e-3c02dd91ad75","Type":"ContainerDied","Data":"f169ce6fbc9e80126e3e9633cf6fc2ec83cff1ef337f8d7db1f1e5a725530883"} Oct 30 01:20:41.838502 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.838181 4927 scope.go:122] "RemoveContainer" containerID="c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce" Oct 30 01:20:41.853982 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.853957 4927 scope.go:122] "RemoveContainer" containerID="c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce" Oct 30 01:20:41.854363 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:41.854312 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce\": container with ID starting with c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce not found: ID does not exist" containerID="c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce" Oct 30 01:20:41.854491 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.854375 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce"} err="failed to get container status \"c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce\": rpc error: code = NotFound desc = could not find container \"c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce\": container with ID starting with c0dc81f382d9daef693e9ae4ddbe191dbf1dab77a0a382cf16ca1d7ea5631dce not found: ID does not exist" Oct 30 01:20:41.908174 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.908141 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dba0780-044e-415d-995e-3c02dd91ad75-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dba0780-044e-415d-995e-3c02dd91ad75-openshift-service-ca\") pod \"3dba0780-044e-415d-995e-3c02dd91ad75\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " Oct 30 01:20:41.908429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.908407 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-maas-api-tls\") pod \"3dba0780-044e-415d-995e-3c02dd91ad75\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " Oct 30 01:20:41.908492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.908458 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-metrics-certs\") pod \"3dba0780-044e-415d-995e-3c02dd91ad75\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " Oct 30 01:20:41.908554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.908515 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dba0780-044e-415d-995e-3c02dd91ad75-kube-api-access-54kkc\" (UniqueName: \"kubernetes.io/projected/3dba0780-044e-415d-995e-3c02dd91ad75-kube-api-access-54kkc\") pod \"3dba0780-044e-415d-995e-3c02dd91ad75\" (UID: \"3dba0780-044e-415d-995e-3c02dd91ad75\") " Oct 30 01:20:41.908668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.908643 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dba0780-044e-415d-995e-3c02dd91ad75-openshift-service-ca" pod "3dba0780-044e-415d-995e-3c02dd91ad75" (UID: "3dba0780-044e-415d-995e-3c02dd91ad75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:41.909255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.909226 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dba0780-044e-415d-995e-3c02dd91ad75-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:41.910961 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.910928 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dba0780-044e-415d-995e-3c02dd91ad75-kube-api-access-54kkc" pod "3dba0780-044e-415d-995e-3c02dd91ad75" (UID: "3dba0780-044e-415d-995e-3c02dd91ad75"). InnerVolumeSpecName "kube-api-access-54kkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:41.911077 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.910991 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-maas-api-tls" pod "3dba0780-044e-415d-995e-3c02dd91ad75" (UID: "3dba0780-044e-415d-995e-3c02dd91ad75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:41.911411 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:41.911383 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-metrics-certs" pod "3dba0780-044e-415d-995e-3c02dd91ad75" (UID: "3dba0780-044e-415d-995e-3c02dd91ad75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:42.010878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:42.010841 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:42.010878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:42.010872 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dba0780-044e-415d-995e-3c02dd91ad75-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:42.010878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:42.010882 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54kkc\" (UniqueName: \"kubernetes.io/projected/3dba0780-044e-415d-995e-3c02dd91ad75-kube-api-access-54kkc\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:42.205968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:42.205928 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw"] Oct 30 01:20:42.210970 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:42.210943 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a42e5-7b45657b8-tqsrw"] Oct 30 01:20:43.574130 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:43.574096 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dba0780-044e-415d-995e-3c02dd91ad75" path="/var/lib/kubelet/pods/3dba0780-044e-415d-995e-3c02dd91ad75/volumes" Oct 30 01:20:45.536371 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:45.536331 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh"] Oct 30 01:20:45.536913 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:45.536623 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" containerID="cri-o://f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28" gracePeriod=30 Oct 30 01:20:45.550247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:45.550214 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n"] Oct 30 01:20:45.550675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:45.550636 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" containerID="cri-o://458f492dbdfe48789f98ebda7d0c5bd456be466b6343b4add8b2e03cad25bbfd" gracePeriod=30 Oct 30 01:20:47.409594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:47.406351 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" podUID="538ea5e7-d6f3-4391-97f6-4f7389bddb83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:20:48.404786 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:48.404741 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" podUID="2677740b-3783-4ff1-8ff8-7c05c34a7ab5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:20:50.877226 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.877196 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:20:50.888177 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888147 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-credential-socket\") pod \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " Oct 30 01:20:50.888310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888206 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-kube-api-access-x42m9\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-kube-api-access-x42m9\") pod \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " Oct 30 01:20:50.888310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888232 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-podinfo\") pod \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " Oct 30 01:20:50.888310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888266 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-token\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-token\") pod \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " Oct 30 01:20:50.888310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888294 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-certs\") pod \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " Oct 30 01:20:50.888551 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888340 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-envoy\") pod \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " Oct 30 01:20:50.888551 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888389 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istiod-ca-cert\") pod \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " Oct 30 01:20:50.888551 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888402 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-credential-socket" pod "4a43e4d1-4dad-4473-82e8-3c51b2bb8753" (UID: "4a43e4d1-4dad-4473-82e8-3c51b2bb8753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:50.888551 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888421 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-data\") pod \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " Oct 30 01:20:50.888794 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888653 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-socket\") pod \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\" (UID: \"4a43e4d1-4dad-4473-82e8-3c51b2bb8753\") " Oct 30 01:20:50.888926 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.888903 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-data" pod "4a43e4d1-4dad-4473-82e8-3c51b2bb8753" (UID: "4a43e4d1-4dad-4473-82e8-3c51b2bb8753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:50.889160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.889115 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-socket" pod "4a43e4d1-4dad-4473-82e8-3c51b2bb8753" (UID: "4a43e4d1-4dad-4473-82e8-3c51b2bb8753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:50.889229 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.889168 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-certs" pod "4a43e4d1-4dad-4473-82e8-3c51b2bb8753" (UID: "4a43e4d1-4dad-4473-82e8-3c51b2bb8753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:50.889296 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.889272 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istiod-ca-cert" pod "4a43e4d1-4dad-4473-82e8-3c51b2bb8753" (UID: "4a43e4d1-4dad-4473-82e8-3c51b2bb8753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:50.889647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.889627 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.889647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.889646 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.889809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.889658 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.889809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.889666 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.889809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.889674 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.890995 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.890882 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-kube-api-access-x42m9" pod "4a43e4d1-4dad-4473-82e8-3c51b2bb8753" (UID: "4a43e4d1-4dad-4473-82e8-3c51b2bb8753"). InnerVolumeSpecName "kube-api-access-x42m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:50.890995 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.890910 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-podinfo" pod "4a43e4d1-4dad-4473-82e8-3c51b2bb8753" (UID: "4a43e4d1-4dad-4473-82e8-3c51b2bb8753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:20:50.890995 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.890932 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-token" pod "4a43e4d1-4dad-4473-82e8-3c51b2bb8753" (UID: "4a43e4d1-4dad-4473-82e8-3c51b2bb8753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:50.892245 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.892222 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-envoy" pod "4a43e4d1-4dad-4473-82e8-3c51b2bb8753" (UID: "4a43e4d1-4dad-4473-82e8-3c51b2bb8753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:50.928349 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.928317 4927 generic.go:356] "Generic (PLEG): container finished" podID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerID="458f492dbdfe48789f98ebda7d0c5bd456be466b6343b4add8b2e03cad25bbfd" exitCode=0 Oct 30 01:20:50.928493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.928400 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" event={"ID":"d41bd266-1e41-448a-bf1c-ca43487dd741","Type":"ContainerDied","Data":"458f492dbdfe48789f98ebda7d0c5bd456be466b6343b4add8b2e03cad25bbfd"} Oct 30 01:20:50.928493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.928459 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" event={"ID":"d41bd266-1e41-448a-bf1c-ca43487dd741","Type":"ContainerDied","Data":"8f2b7f1264532cda08e537f2de4104d65effa74d0b95343635536160d86dea68"} Oct 30 01:20:50.928493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.928481 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f2b7f1264532cda08e537f2de4104d65effa74d0b95343635536160d86dea68" Oct 30 01:20:50.929985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.929962 4927 generic.go:356] "Generic (PLEG): container finished" podID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerID="f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28" exitCode=0 Oct 30 01:20:50.930110 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.930013 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" Oct 30 01:20:50.930110 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.930042 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" event={"ID":"4a43e4d1-4dad-4473-82e8-3c51b2bb8753","Type":"ContainerDied","Data":"f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28"} Oct 30 01:20:50.930110 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.930090 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh" event={"ID":"4a43e4d1-4dad-4473-82e8-3c51b2bb8753","Type":"ContainerDied","Data":"f1e84b4a70134f1a32881f9b4b899d93997ce0f2fe685686efca2c518197dd0c"} Oct 30 01:20:50.930271 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.930119 4927 scope.go:122] "RemoveContainer" containerID="f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28" Oct 30 01:20:50.931727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.931705 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:20:50.946193 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.946168 4927 scope.go:122] "RemoveContainer" containerID="f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28" Oct 30 01:20:50.946548 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:50.946522 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28\": container with ID starting with f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28 not found: ID does not exist" containerID="f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28" Oct 30 01:20:50.946668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.946557 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28"} err="failed to get container status \"f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28\": rpc error: code = NotFound desc = could not find container \"f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28\": container with ID starting with f0d2cc30ac8d78e32a022755f7d1dc042976daba3c3e45c6209e28880e101f28 not found: ID does not exist" Oct 30 01:20:50.990792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.990753 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-kube-api-access-scbj7\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-kube-api-access-scbj7\") pod \"d41bd266-1e41-448a-bf1c-ca43487dd741\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " Oct 30 01:20:50.990964 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.990809 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-envoy\") pod \"d41bd266-1e41-448a-bf1c-ca43487dd741\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " Oct 30 01:20:50.991113 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.991087 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-credential-socket\") pod \"d41bd266-1e41-448a-bf1c-ca43487dd741\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " Oct 30 01:20:50.991189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.991155 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-data\") pod \"d41bd266-1e41-448a-bf1c-ca43487dd741\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " Oct 30 01:20:50.991297 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.991268 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-podinfo\") pod \"d41bd266-1e41-448a-bf1c-ca43487dd741\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " Oct 30 01:20:50.991414 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.991319 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-socket\") pod \"d41bd266-1e41-448a-bf1c-ca43487dd741\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " Oct 30 01:20:50.991414 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.991356 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d41bd266-1e41-448a-bf1c-ca43487dd741-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41bd266-1e41-448a-bf1c-ca43487dd741-istiod-ca-cert\") pod \"d41bd266-1e41-448a-bf1c-ca43487dd741\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " Oct 30 01:20:50.991564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.991412 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-certs\") pod \"d41bd266-1e41-448a-bf1c-ca43487dd741\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " Oct 30 01:20:50.991564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.991459 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-token\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-token\") pod \"d41bd266-1e41-448a-bf1c-ca43487dd741\" (UID: \"d41bd266-1e41-448a-bf1c-ca43487dd741\") " Oct 30 01:20:50.992294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.992267 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42m9\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-kube-api-access-x42m9\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.992471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.992297 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.992471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.992311 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.992471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.992324 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43e4d1-4dad-4473-82e8-3c51b2bb8753-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.994155 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.994127 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-data" pod "d41bd266-1e41-448a-bf1c-ca43487dd741" (UID: "d41bd266-1e41-448a-bf1c-ca43487dd741"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:50.995046 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.995025 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-socket" pod "d41bd266-1e41-448a-bf1c-ca43487dd741" (UID: "d41bd266-1e41-448a-bf1c-ca43487dd741"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:50.995461 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.995198 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-credential-socket" pod "d41bd266-1e41-448a-bf1c-ca43487dd741" (UID: "d41bd266-1e41-448a-bf1c-ca43487dd741"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:50.995461 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.995372 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-certs" pod "d41bd266-1e41-448a-bf1c-ca43487dd741" (UID: "d41bd266-1e41-448a-bf1c-ca43487dd741"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:50.995461 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:50.995428 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41bd266-1e41-448a-bf1c-ca43487dd741-istiod-ca-cert" pod "d41bd266-1e41-448a-bf1c-ca43487dd741" (UID: "d41bd266-1e41-448a-bf1c-ca43487dd741"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:51.000951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.000914 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-envoy" pod "d41bd266-1e41-448a-bf1c-ca43487dd741" (UID: "d41bd266-1e41-448a-bf1c-ca43487dd741"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:51.001078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.001049 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-kube-api-access-scbj7" pod "d41bd266-1e41-448a-bf1c-ca43487dd741" (UID: "d41bd266-1e41-448a-bf1c-ca43487dd741"). InnerVolumeSpecName "kube-api-access-scbj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:51.001501 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.001476 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-token" pod "d41bd266-1e41-448a-bf1c-ca43487dd741" (UID: "d41bd266-1e41-448a-bf1c-ca43487dd741"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:51.002647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.002618 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-podinfo" pod "d41bd266-1e41-448a-bf1c-ca43487dd741" (UID: "d41bd266-1e41-448a-bf1c-ca43487dd741"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:20:51.036924 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.036882 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh"] Oct 30 01:20:51.045093 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.045067 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-7sjxh"] Oct 30 01:20:51.111363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.099442 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.111363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.099481 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41bd266-1e41-448a-bf1c-ca43487dd741-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.111363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.099501 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.111363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.099514 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.111363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.099528 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scbj7\" (UniqueName: \"kubernetes.io/projected/d41bd266-1e41-448a-bf1c-ca43487dd741-kube-api-access-scbj7\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.111363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.099543 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.111363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.099555 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.111363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.099585 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.111363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.099600 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41bd266-1e41-448a-bf1c-ca43487dd741-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:51.575724 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.575689 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" path="/var/lib/kubelet/pods/4a43e4d1-4dad-4473-82e8-3c51b2bb8753/volumes" Oct 30 01:20:51.940602 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.940555 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n" Oct 30 01:20:51.994405 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:51.994370 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n"] Oct 30 01:20:52.007312 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:52.007280 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-npq7n"] Oct 30 01:20:53.575660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:53.575627 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" path="/var/lib/kubelet/pods/d41bd266-1e41-448a-bf1c-ca43487dd741/volumes" Oct 30 01:20:54.716620 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.715310 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 01:20:54.780128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.780088 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-plugins-config-volume\") pod \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\" (UID: \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\") " Oct 30 01:20:54.780319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.780175 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-kube-api-access-r6lvk\" (UniqueName: \"kubernetes.io/projected/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-kube-api-access-r6lvk\") pod \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\" (UID: \"2677740b-3783-4ff1-8ff8-7c05c34a7ab5\") " Oct 30 01:20:54.785744 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.785700 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-plugins-config-volume" pod "2677740b-3783-4ff1-8ff8-7c05c34a7ab5" (UID: "2677740b-3783-4ff1-8ff8-7c05c34a7ab5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:54.791849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.791821 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-kube-api-access-r6lvk" pod "2677740b-3783-4ff1-8ff8-7c05c34a7ab5" (UID: "2677740b-3783-4ff1-8ff8-7c05c34a7ab5"). InnerVolumeSpecName "kube-api-access-r6lvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:54.830790 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.830761 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 01:20:54.883209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.883176 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6lvk\" (UniqueName: \"kubernetes.io/projected/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-kube-api-access-r6lvk\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:54.883209 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.883208 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2677740b-3783-4ff1-8ff8-7c05c34a7ab5-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:54.967893 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.967806 4927 generic.go:356] "Generic (PLEG): container finished" podID="2677740b-3783-4ff1-8ff8-7c05c34a7ab5" containerID="2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85" exitCode=0 Oct 30 01:20:54.967893 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.967877 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" Oct 30 01:20:54.968141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.967897 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" event={"ID":"2677740b-3783-4ff1-8ff8-7c05c34a7ab5","Type":"ContainerDied","Data":"2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85"} Oct 30 01:20:54.968141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.967955 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92" event={"ID":"2677740b-3783-4ff1-8ff8-7c05c34a7ab5","Type":"ContainerDied","Data":"2a69ec5e1cfb0e7ef88e5634a89361bc982118cda573bad444757db86fbb090c"} Oct 30 01:20:54.968141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.967978 4927 scope.go:122] "RemoveContainer" containerID="2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85" Oct 30 01:20:54.969849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.969645 4927 generic.go:356] "Generic (PLEG): container finished" podID="538ea5e7-d6f3-4391-97f6-4f7389bddb83" containerID="6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087" exitCode=0 Oct 30 01:20:54.969849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.969816 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" event={"ID":"538ea5e7-d6f3-4391-97f6-4f7389bddb83","Type":"ContainerDied","Data":"6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087"} Oct 30 01:20:54.970059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.969900 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" event={"ID":"538ea5e7-d6f3-4391-97f6-4f7389bddb83","Type":"ContainerDied","Data":"6c0831ea8fc3521e3c0c8402481b09d8a1c00593e68e484f30f0f5e616b7b0a0"} Oct 30 01:20:54.970059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.970010 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb" Oct 30 01:20:54.981421 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.981402 4927 scope.go:122] "RemoveContainer" containerID="2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85" Oct 30 01:20:54.981764 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:54.981743 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85\": container with ID starting with 2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85 not found: ID does not exist" containerID="2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85" Oct 30 01:20:54.981857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.981770 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85"} err="failed to get container status \"2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85\": rpc error: code = NotFound desc = could not find container \"2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85\": container with ID starting with 2f26095b27d1990e57d3d7c78bb7bcbdb9284c6a92f066e90f0bada281b62f85 not found: ID does not exist" Oct 30 01:20:54.981857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.981787 4927 scope.go:122] "RemoveContainer" containerID="6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087" Oct 30 01:20:54.984388 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.984364 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538ea5e7-d6f3-4391-97f6-4f7389bddb83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538ea5e7-d6f3-4391-97f6-4f7389bddb83-plugins-config-volume\") pod \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\" (UID: \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\") " Oct 30 01:20:54.984525 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.984508 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538ea5e7-d6f3-4391-97f6-4f7389bddb83-kube-api-access-dq5tr\" (UniqueName: \"kubernetes.io/projected/538ea5e7-d6f3-4391-97f6-4f7389bddb83-kube-api-access-dq5tr\") pod \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\" (UID: \"538ea5e7-d6f3-4391-97f6-4f7389bddb83\") " Oct 30 01:20:54.984732 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.984713 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538ea5e7-d6f3-4391-97f6-4f7389bddb83-plugins-config-volume" pod "538ea5e7-d6f3-4391-97f6-4f7389bddb83" (UID: "538ea5e7-d6f3-4391-97f6-4f7389bddb83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:54.985395 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.985370 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538ea5e7-d6f3-4391-97f6-4f7389bddb83-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:54.986664 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.986636 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538ea5e7-d6f3-4391-97f6-4f7389bddb83-kube-api-access-dq5tr" pod "538ea5e7-d6f3-4391-97f6-4f7389bddb83" (UID: "538ea5e7-d6f3-4391-97f6-4f7389bddb83"). InnerVolumeSpecName "kube-api-access-dq5tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:54.992773 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.992754 4927 scope.go:122] "RemoveContainer" containerID="6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087" Oct 30 01:20:54.993037 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:20:54.993021 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087\": container with ID starting with 6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087 not found: ID does not exist" containerID="6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087" Oct 30 01:20:54.993095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:54.993046 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087"} err="failed to get container status \"6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087\": rpc error: code = NotFound desc = could not find container \"6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087\": container with ID starting with 6afe794b27613573c67f72744118b3714b397e67fa6555f73f8c91aed43eb087 not found: ID does not exist" Oct 30 01:20:55.023644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:55.023609 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92"] Oct 30 01:20:55.029368 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:55.029334 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b3593a-58cf9fd4c8-t6j92"] Oct 30 01:20:55.097584 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:55.096751 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5tr\" (UniqueName: \"kubernetes.io/projected/538ea5e7-d6f3-4391-97f6-4f7389bddb83-kube-api-access-dq5tr\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.338195 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:55.338156 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb"] Oct 30 01:20:55.353049 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:55.353012 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b3593a-5465ffb8c9-7gpkb"] Oct 30 01:20:55.361417 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:55.361385 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc"] Oct 30 01:20:55.361666 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:55.361639 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" containerID="cri-o://37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0" gracePeriod=30 Oct 30 01:20:55.677626 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:55.675298 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2677740b-3783-4ff1-8ff8-7c05c34a7ab5" path="/var/lib/kubelet/pods/2677740b-3783-4ff1-8ff8-7c05c34a7ab5/volumes" Oct 30 01:20:55.677626 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:20:55.675928 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538ea5e7-d6f3-4391-97f6-4f7389bddb83" path="/var/lib/kubelet/pods/538ea5e7-d6f3-4391-97f6-4f7389bddb83/volumes" Oct 30 01:21:00.704295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.704263 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:21:00.743774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.743742 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-socket\") pod \"667e312d-afd1-4032-909f-aa766af53204\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " Oct 30 01:21:00.743930 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.743791 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/667e312d-afd1-4032-909f-aa766af53204-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e312d-afd1-4032-909f-aa766af53204-istiod-ca-cert\") pod \"667e312d-afd1-4032-909f-aa766af53204\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " Oct 30 01:21:00.743930 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.743823 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-certs\") pod \"667e312d-afd1-4032-909f-aa766af53204\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " Oct 30 01:21:00.743930 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.743874 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-envoy\") pod \"667e312d-afd1-4032-909f-aa766af53204\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " Oct 30 01:21:00.744111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.743991 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-credential-socket\") pod \"667e312d-afd1-4032-909f-aa766af53204\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " Oct 30 01:21:00.744111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.744047 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-istio-token\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-istio-token\") pod \"667e312d-afd1-4032-909f-aa766af53204\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " Oct 30 01:21:00.744111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.744064 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-socket" pod "667e312d-afd1-4032-909f-aa766af53204" (UID: "667e312d-afd1-4032-909f-aa766af53204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:00.744111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.744095 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-data\") pod \"667e312d-afd1-4032-909f-aa766af53204\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " Oct 30 01:21:00.744361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.744127 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/667e312d-afd1-4032-909f-aa766af53204-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e312d-afd1-4032-909f-aa766af53204-istio-podinfo\") pod \"667e312d-afd1-4032-909f-aa766af53204\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " Oct 30 01:21:00.744361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.744197 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-kube-api-access-m9gh2\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-kube-api-access-m9gh2\") pod \"667e312d-afd1-4032-909f-aa766af53204\" (UID: \"667e312d-afd1-4032-909f-aa766af53204\") " Oct 30 01:21:00.744361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.744272 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-credential-socket" pod "667e312d-afd1-4032-909f-aa766af53204" (UID: "667e312d-afd1-4032-909f-aa766af53204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:00.744523 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.744384 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667e312d-afd1-4032-909f-aa766af53204-istiod-ca-cert" pod "667e312d-afd1-4032-909f-aa766af53204" (UID: "667e312d-afd1-4032-909f-aa766af53204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:00.744965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.744944 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-data" pod "667e312d-afd1-4032-909f-aa766af53204" (UID: "667e312d-afd1-4032-909f-aa766af53204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:00.745436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.745257 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:00.745436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.745281 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:00.745436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.745294 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:00.745436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.745307 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e312d-afd1-4032-909f-aa766af53204-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:00.745436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.745360 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-certs" pod "667e312d-afd1-4032-909f-aa766af53204" (UID: "667e312d-afd1-4032-909f-aa766af53204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:00.755074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.754901 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/667e312d-afd1-4032-909f-aa766af53204-istio-podinfo" pod "667e312d-afd1-4032-909f-aa766af53204" (UID: "667e312d-afd1-4032-909f-aa766af53204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:00.755074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.754943 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-envoy" pod "667e312d-afd1-4032-909f-aa766af53204" (UID: "667e312d-afd1-4032-909f-aa766af53204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:00.755074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.755034 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-istio-token" pod "667e312d-afd1-4032-909f-aa766af53204" (UID: "667e312d-afd1-4032-909f-aa766af53204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:00.756761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.756739 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-kube-api-access-m9gh2" pod "667e312d-afd1-4032-909f-aa766af53204" (UID: "667e312d-afd1-4032-909f-aa766af53204"). InnerVolumeSpecName "kube-api-access-m9gh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:00.846774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.846741 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:00.846774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.846773 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e312d-afd1-4032-909f-aa766af53204-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:00.846977 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.846784 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:00.846977 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.846795 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e312d-afd1-4032-909f-aa766af53204-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:00.846977 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:00.846806 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9gh2\" (UniqueName: \"kubernetes.io/projected/667e312d-afd1-4032-909f-aa766af53204-kube-api-access-m9gh2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:01.041866 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.041784 4927 generic.go:356] "Generic (PLEG): container finished" podID="667e312d-afd1-4032-909f-aa766af53204" containerID="37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0" exitCode=0 Oct 30 01:21:01.041866 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.041844 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" Oct 30 01:21:01.042142 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.041865 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" event={"ID":"667e312d-afd1-4032-909f-aa766af53204","Type":"ContainerDied","Data":"37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0"} Oct 30 01:21:01.042142 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.041944 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc" event={"ID":"667e312d-afd1-4032-909f-aa766af53204","Type":"ContainerDied","Data":"7155aa3ed96f26e647582fcb104ac5e3091047ccd5efd79d3fc8722a485ee302"} Oct 30 01:21:01.042142 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.041966 4927 scope.go:122] "RemoveContainer" containerID="37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0" Oct 30 01:21:01.058658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.058632 4927 scope.go:122] "RemoveContainer" containerID="37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0" Oct 30 01:21:01.058966 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:01.058946 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0\": container with ID starting with 37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0 not found: ID does not exist" containerID="37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0" Oct 30 01:21:01.059021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.058975 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0"} err="failed to get container status \"37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0\": rpc error: code = NotFound desc = could not find container \"37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0\": container with ID starting with 37892aaf1d0f992740b747d4c4d0cd69e6ff0b34f12c4f4bf59285b5cc1058c0 not found: ID does not exist" Oct 30 01:21:01.094442 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.094389 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc"] Oct 30 01:21:01.105733 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.105711 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-pgbtc"] Oct 30 01:21:01.186151 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.186117 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl"] Oct 30 01:21:01.219826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.219795 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk"] Oct 30 01:21:01.631444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:01.627083 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667e312d-afd1-4032-909f-aa766af53204" path="/var/lib/kubelet/pods/667e312d-afd1-4032-909f-aa766af53204/volumes" Oct 30 01:21:04.444444 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:04.444401 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:21:06.207631 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:06.207558 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" podUID="816fb468-458b-4597-8da0-5fe17e7dc3d8" containerName="payload-processing" containerID="cri-o://113ee3b7c5940e75ef3bf634725b06eaa644fa67fc71893cb345ae4ce4565b01" gracePeriod=25 Oct 30 01:21:06.220701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:06.220655 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" podUID="c407565c-0ba4-4410-bedc-e6c9d4e5afe7" containerName="payload-pre-processing" containerID="cri-o://b206851fdd39536d3189975be76cf7bc1ce01e075e965fcf1e049ea26a52fb76" gracePeriod=25 Oct 30 01:21:06.897595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:06.896869 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" podUID="c407565c-0ba4-4410-bedc-e6c9d4e5afe7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:06.905591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:06.901688 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" podUID="816fb468-458b-4597-8da0-5fe17e7dc3d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:07.016313 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.016274 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5"] Oct 30 01:21:07.016530 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.016476 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" podUID="79b8fa70-f329-4ae1-8384-068f7fd1d151" containerName="maas-api" containerID="cri-o://c90f075f13be45740bd6b95272508185ebc402d8a5ffc85e59516cf32120ddfa" gracePeriod=30 Oct 30 01:21:07.170565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.170474 4927 generic.go:356] "Generic (PLEG): container finished" podID="79b8fa70-f329-4ae1-8384-068f7fd1d151" containerID="c90f075f13be45740bd6b95272508185ebc402d8a5ffc85e59516cf32120ddfa" exitCode=0 Oct 30 01:21:07.170565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.170546 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" event={"ID":"79b8fa70-f329-4ae1-8384-068f7fd1d151","Type":"ContainerDied","Data":"c90f075f13be45740bd6b95272508185ebc402d8a5ffc85e59516cf32120ddfa"} Oct 30 01:21:07.764684 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.761605 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 01:21:07.912445 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.912401 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-maas-api-tls\") pod \"79b8fa70-f329-4ae1-8384-068f7fd1d151\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " Oct 30 01:21:07.912789 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.912762 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b8fa70-f329-4ae1-8384-068f7fd1d151-kube-api-access-qqdn7\" (UniqueName: \"kubernetes.io/projected/79b8fa70-f329-4ae1-8384-068f7fd1d151-kube-api-access-qqdn7\") pod \"79b8fa70-f329-4ae1-8384-068f7fd1d151\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " Oct 30 01:21:07.912899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.912885 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b8fa70-f329-4ae1-8384-068f7fd1d151-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8fa70-f329-4ae1-8384-068f7fd1d151-openshift-service-ca\") pod \"79b8fa70-f329-4ae1-8384-068f7fd1d151\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " Oct 30 01:21:07.912969 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.912907 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-metrics-certs\") pod \"79b8fa70-f329-4ae1-8384-068f7fd1d151\" (UID: \"79b8fa70-f329-4ae1-8384-068f7fd1d151\") " Oct 30 01:21:07.913498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.913405 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b8fa70-f329-4ae1-8384-068f7fd1d151-openshift-service-ca" pod "79b8fa70-f329-4ae1-8384-068f7fd1d151" (UID: "79b8fa70-f329-4ae1-8384-068f7fd1d151"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:07.913891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.913861 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b8fa70-f329-4ae1-8384-068f7fd1d151-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:07.915820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.915780 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-metrics-certs" pod "79b8fa70-f329-4ae1-8384-068f7fd1d151" (UID: "79b8fa70-f329-4ae1-8384-068f7fd1d151"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:07.916092 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.916068 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-maas-api-tls" pod "79b8fa70-f329-4ae1-8384-068f7fd1d151" (UID: "79b8fa70-f329-4ae1-8384-068f7fd1d151"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:07.916182 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:07.916115 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b8fa70-f329-4ae1-8384-068f7fd1d151-kube-api-access-qqdn7" pod "79b8fa70-f329-4ae1-8384-068f7fd1d151" (UID: "79b8fa70-f329-4ae1-8384-068f7fd1d151"). InnerVolumeSpecName "kube-api-access-qqdn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:08.027038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:08.023116 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqdn7\" (UniqueName: \"kubernetes.io/projected/79b8fa70-f329-4ae1-8384-068f7fd1d151-kube-api-access-qqdn7\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:08.027038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:08.023160 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:08.027038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:08.023174 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b8fa70-f329-4ae1-8384-068f7fd1d151-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:08.205089 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:08.205057 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" Oct 30 01:21:08.205089 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:08.205070 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5" event={"ID":"79b8fa70-f329-4ae1-8384-068f7fd1d151","Type":"ContainerDied","Data":"0fd927cdfb17a54d985b8e5b86795a17ba5d9a78112109038c83c9488eb9c800"} Oct 30 01:21:08.205335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:08.205137 4927 scope.go:122] "RemoveContainer" containerID="c90f075f13be45740bd6b95272508185ebc402d8a5ffc85e59516cf32120ddfa" Oct 30 01:21:08.265051 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:08.265022 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5"] Oct 30 01:21:08.287393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:08.287249 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bea3ef-79787b74f5-glkx5"] Oct 30 01:21:08.659196 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:08.659160 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v"] Oct 30 01:21:09.599294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:09.599250 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b8fa70-f329-4ae1-8384-068f7fd1d151" path="/var/lib/kubelet/pods/79b8fa70-f329-4ae1-8384-068f7fd1d151/volumes" Oct 30 01:21:13.660111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:13.660063 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" podUID="07e63105-a131-4eb5-8055-e6dcc033c00b" containerName="payload-pre-processing" containerID="cri-o://1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8" gracePeriod=25 Oct 30 01:21:14.027833 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.027801 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:21:14.198635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.198516 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e63105-a131-4eb5-8055-e6dcc033c00b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e63105-a131-4eb5-8055-e6dcc033c00b-plugins-config-volume\") pod \"07e63105-a131-4eb5-8055-e6dcc033c00b\" (UID: \"07e63105-a131-4eb5-8055-e6dcc033c00b\") " Oct 30 01:21:14.198888 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.198857 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e63105-a131-4eb5-8055-e6dcc033c00b-kube-api-access-cxf7t\" (UniqueName: \"kubernetes.io/projected/07e63105-a131-4eb5-8055-e6dcc033c00b-kube-api-access-cxf7t\") pod \"07e63105-a131-4eb5-8055-e6dcc033c00b\" (UID: \"07e63105-a131-4eb5-8055-e6dcc033c00b\") " Oct 30 01:21:14.199016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.198892 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e63105-a131-4eb5-8055-e6dcc033c00b-plugins-config-volume" pod "07e63105-a131-4eb5-8055-e6dcc033c00b" (UID: "07e63105-a131-4eb5-8055-e6dcc033c00b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:14.199502 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.199481 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e63105-a131-4eb5-8055-e6dcc033c00b-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:14.201066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.201043 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e63105-a131-4eb5-8055-e6dcc033c00b-kube-api-access-cxf7t" pod "07e63105-a131-4eb5-8055-e6dcc033c00b" (UID: "07e63105-a131-4eb5-8055-e6dcc033c00b"). InnerVolumeSpecName "kube-api-access-cxf7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:14.262718 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.262687 4927 generic.go:356] "Generic (PLEG): container finished" podID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerID="35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02" exitCode=0 Oct 30 01:21:14.262868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.262761 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" event={"ID":"1038975e-5fd3-4674-9374-e4a46799b7ff","Type":"ContainerDied","Data":"35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02"} Oct 30 01:21:14.263277 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.263250 4927 scope.go:122] "RemoveContainer" containerID="35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02" Oct 30 01:21:14.264322 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.264296 4927 generic.go:356] "Generic (PLEG): container finished" podID="07e63105-a131-4eb5-8055-e6dcc033c00b" containerID="1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8" exitCode=0 Oct 30 01:21:14.264418 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.264357 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" Oct 30 01:21:14.264418 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.264412 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" event={"ID":"07e63105-a131-4eb5-8055-e6dcc033c00b","Type":"ContainerDied","Data":"1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8"} Oct 30 01:21:14.264523 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.264445 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v" event={"ID":"07e63105-a131-4eb5-8055-e6dcc033c00b","Type":"ContainerDied","Data":"e8d45e11388788f587d132684817fc4c5a68cf734a1f99695e72bb1c4eccb667"} Oct 30 01:21:14.264523 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.264466 4927 scope.go:122] "RemoveContainer" containerID="1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8" Oct 30 01:21:14.278588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.266995 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:21:14.278588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.267210 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:21:14.278588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.267494 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:21:14.295043 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.295022 4927 scope.go:122] "RemoveContainer" containerID="1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8" Oct 30 01:21:14.295392 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:14.295366 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8\": container with ID starting with 1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8 not found: ID does not exist" containerID="1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8" Oct 30 01:21:14.295488 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.295403 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8"} err="failed to get container status \"1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8\": rpc error: code = NotFound desc = could not find container \"1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8\": container with ID starting with 1cd1433c6a666ed9b4586028dc612289b8819507594bd80cc559a51a9e063ca8 not found: ID does not exist" Oct 30 01:21:14.301317 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.301288 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxf7t\" (UniqueName: \"kubernetes.io/projected/07e63105-a131-4eb5-8055-e6dcc033c00b-kube-api-access-cxf7t\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:14.358496 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.358468 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v"] Oct 30 01:21:14.364071 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:14.364044 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-74998646b7-szq9v"] Oct 30 01:21:15.294595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:15.278956 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" event={"ID":"1038975e-5fd3-4674-9374-e4a46799b7ff","Type":"ContainerStarted","Data":"6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb"} Oct 30 01:21:15.406126 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:15.406088 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8"] Oct 30 01:21:15.406335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:15.406290 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" containerID="cri-o://4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4" gracePeriod=30 Oct 30 01:21:15.696208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:15.696167 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e63105-a131-4eb5-8055-e6dcc033c00b" path="/var/lib/kubelet/pods/07e63105-a131-4eb5-8055-e6dcc033c00b/volumes" Oct 30 01:21:15.711601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:15.709501 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:21:15.711601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:15.709643 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:21:15.711601 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:15.709698 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 01:21:16.710389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:16.710353 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:21:16.710800 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:16.710412 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 01:21:16.864433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:16.864388 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" podUID="816fb468-458b-4597-8da0-5fe17e7dc3d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:16.864674 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:16.864497 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" podUID="c407565c-0ba4-4410-bedc-e6c9d4e5afe7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:17.710363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:17.710325 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:21:17.710528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:17.710390 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 01:21:18.709827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:18.709791 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:21:18.710039 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:18.709881 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 01:21:19.710255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:19.710216 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:21:19.710780 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:19.710288 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 01:21:20.729524 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.729492 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:21:20.757488 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.757459 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:21:20.757871 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.757796 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:21:20.760030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.760010 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:21:20.874595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.873898 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-token\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-token\") pod \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " Oct 30 01:21:20.874595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.873984 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-data\") pod \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " Oct 30 01:21:20.874595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.874140 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-certs\") pod \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " Oct 30 01:21:20.874595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.874217 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-envoy\") pod \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " Oct 30 01:21:20.874595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.874248 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-podinfo\") pod \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " Oct 30 01:21:20.874595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.874401 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-credential-socket\") pod \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " Oct 30 01:21:20.874595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.874496 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istiod-ca-cert\") pod \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " Oct 30 01:21:20.875128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.874634 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-socket\") pod \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " Oct 30 01:21:20.875128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.874682 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-kube-api-access-66zfd\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-kube-api-access-66zfd\") pod \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\" (UID: \"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d\") " Oct 30 01:21:20.881642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.878215 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-socket" pod "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" (UID: "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:20.881642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.878363 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-data" pod "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" (UID: "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:20.881642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.878547 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-certs" pod "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" (UID: "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:20.881642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.878770 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-credential-socket" pod "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" (UID: "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:20.881642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.878886 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istiod-ca-cert" pod "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" (UID: "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:20.886854 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.886813 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-podinfo" pod "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" (UID: "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:20.887945 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.887906 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-kube-api-access-66zfd" pod "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" (UID: "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d"). InnerVolumeSpecName "kube-api-access-66zfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:20.889067 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.889040 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-token" pod "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" (UID: "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:20.889528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.889492 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-envoy" pod "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" (UID: "f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:20.977453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.977373 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:20.977453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.977405 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:20.977453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.977416 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:20.977453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.977425 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:20.977453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.977435 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:20.977453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.977443 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:20.977453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.977451 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66zfd\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-kube-api-access-66zfd\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:20.977813 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.977459 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:20.977813 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:20.977470 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:21.382851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.382812 4927 generic.go:356] "Generic (PLEG): container finished" podID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerID="4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4" exitCode=0 Oct 30 01:21:21.384253 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.384213 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" Oct 30 01:21:21.385697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.385667 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" event={"ID":"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d","Type":"ContainerDied","Data":"4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4"} Oct 30 01:21:21.385915 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.385893 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8" event={"ID":"f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d","Type":"ContainerDied","Data":"1b0fa67ca24acf973fc6be98178fb9edb92df075f37d5df09820aa16769a9afc"} Oct 30 01:21:21.386109 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.386093 4927 scope.go:122] "RemoveContainer" containerID="4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4" Oct 30 01:21:21.402359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.402335 4927 scope.go:122] "RemoveContainer" containerID="4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4" Oct 30 01:21:21.402688 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:21.402665 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4\": container with ID starting with 4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4 not found: ID does not exist" containerID="4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4" Oct 30 01:21:21.402751 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.402697 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4"} err="failed to get container status \"4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4\": rpc error: code = NotFound desc = could not find container \"4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4\": container with ID starting with 4488e9a1d642eb1d516a2606c0116b45b5506bdd61dc485c960a766cca40a9a4 not found: ID does not exist" Oct 30 01:21:21.448467 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.448440 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8"] Oct 30 01:21:21.460035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.460007 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-p4np8"] Oct 30 01:21:21.575562 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:21.575531 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" path="/var/lib/kubelet/pods/f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d/volumes" Oct 30 01:21:25.600383 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:25.600351 4927 scope.go:122] "RemoveContainer" containerID="458f492dbdfe48789f98ebda7d0c5bd456be466b6343b4add8b2e03cad25bbfd" Oct 30 01:21:26.490555 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.490526 4927 generic.go:356] "Generic (PLEG): container finished" podID="816fb468-458b-4597-8da0-5fe17e7dc3d8" containerID="113ee3b7c5940e75ef3bf634725b06eaa644fa67fc71893cb345ae4ce4565b01" exitCode=0 Oct 30 01:21:26.490766 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.490734 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" event={"ID":"816fb468-458b-4597-8da0-5fe17e7dc3d8","Type":"ContainerDied","Data":"113ee3b7c5940e75ef3bf634725b06eaa644fa67fc71893cb345ae4ce4565b01"} Oct 30 01:21:26.492935 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.492804 4927 generic.go:356] "Generic (PLEG): container finished" podID="c407565c-0ba4-4410-bedc-e6c9d4e5afe7" containerID="b206851fdd39536d3189975be76cf7bc1ce01e075e965fcf1e049ea26a52fb76" exitCode=0 Oct 30 01:21:26.492935 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.492863 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" event={"ID":"c407565c-0ba4-4410-bedc-e6c9d4e5afe7","Type":"ContainerDied","Data":"b206851fdd39536d3189975be76cf7bc1ce01e075e965fcf1e049ea26a52fb76"} Oct 30 01:21:26.666087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.666058 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 01:21:26.688687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.688661 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 01:21:26.713047 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.713019 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816fb468-458b-4597-8da0-5fe17e7dc3d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816fb468-458b-4597-8da0-5fe17e7dc3d8-plugins-config-volume\") pod \"816fb468-458b-4597-8da0-5fe17e7dc3d8\" (UID: \"816fb468-458b-4597-8da0-5fe17e7dc3d8\") " Oct 30 01:21:26.713177 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.713143 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-plugins-config-volume\") pod \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\" (UID: \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\") " Oct 30 01:21:26.713219 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.713213 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816fb468-458b-4597-8da0-5fe17e7dc3d8-kube-api-access-fz49g\" (UniqueName: \"kubernetes.io/projected/816fb468-458b-4597-8da0-5fe17e7dc3d8-kube-api-access-fz49g\") pod \"816fb468-458b-4597-8da0-5fe17e7dc3d8\" (UID: \"816fb468-458b-4597-8da0-5fe17e7dc3d8\") " Oct 30 01:21:26.713327 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.713316 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-kube-api-access-92ssw\" (UniqueName: \"kubernetes.io/projected/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-kube-api-access-92ssw\") pod \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\" (UID: \"c407565c-0ba4-4410-bedc-e6c9d4e5afe7\") " Oct 30 01:21:26.713527 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.713493 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816fb468-458b-4597-8da0-5fe17e7dc3d8-plugins-config-volume" pod "816fb468-458b-4597-8da0-5fe17e7dc3d8" (UID: "816fb468-458b-4597-8da0-5fe17e7dc3d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:26.713636 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.713542 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-plugins-config-volume" pod "c407565c-0ba4-4410-bedc-e6c9d4e5afe7" (UID: "c407565c-0ba4-4410-bedc-e6c9d4e5afe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:26.713870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.713854 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816fb468-458b-4597-8da0-5fe17e7dc3d8-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:26.713925 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.713873 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:26.715443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.715418 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-kube-api-access-92ssw" pod "c407565c-0ba4-4410-bedc-e6c9d4e5afe7" (UID: "c407565c-0ba4-4410-bedc-e6c9d4e5afe7"). InnerVolumeSpecName "kube-api-access-92ssw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:26.715605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.715557 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816fb468-458b-4597-8da0-5fe17e7dc3d8-kube-api-access-fz49g" pod "816fb468-458b-4597-8da0-5fe17e7dc3d8" (UID: "816fb468-458b-4597-8da0-5fe17e7dc3d8"). InnerVolumeSpecName "kube-api-access-fz49g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:26.816129 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.816053 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz49g\" (UniqueName: \"kubernetes.io/projected/816fb468-458b-4597-8da0-5fe17e7dc3d8-kube-api-access-fz49g\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:26.816129 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:26.816080 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92ssw\" (UniqueName: \"kubernetes.io/projected/c407565c-0ba4-4410-bedc-e6c9d4e5afe7-kube-api-access-92ssw\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:27.502639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.502522 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" Oct 30 01:21:27.502639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.502539 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk" event={"ID":"c407565c-0ba4-4410-bedc-e6c9d4e5afe7","Type":"ContainerDied","Data":"64affd99a046d8ae31b39ff1b76ec69868b169fbc3a87752bd21e12572f7751e"} Oct 30 01:21:27.502875 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.502654 4927 scope.go:122] "RemoveContainer" containerID="b206851fdd39536d3189975be76cf7bc1ce01e075e965fcf1e049ea26a52fb76" Oct 30 01:21:27.504426 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.504406 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" Oct 30 01:21:27.504640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.504428 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl" event={"ID":"816fb468-458b-4597-8da0-5fe17e7dc3d8","Type":"ContainerDied","Data":"a0ca072e7d903898cf780b4c259198e1da0fa853997ec47b84578f0daaa98792"} Oct 30 01:21:27.514444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.514403 4927 scope.go:122] "RemoveContainer" containerID="113ee3b7c5940e75ef3bf634725b06eaa644fa67fc71893cb345ae4ce4565b01" Oct 30 01:21:27.555127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.555098 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk"] Oct 30 01:21:27.561165 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.561142 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bea3ef-7d644dff79-2mmvk"] Oct 30 01:21:27.573041 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.573017 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c407565c-0ba4-4410-bedc-e6c9d4e5afe7" path="/var/lib/kubelet/pods/c407565c-0ba4-4410-bedc-e6c9d4e5afe7/volumes" Oct 30 01:21:27.597519 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.597493 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl"] Oct 30 01:21:27.604500 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:27.604476 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bea3ef-5dc5b8c9b4-7tdcl"] Oct 30 01:21:29.574061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.574029 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816fb468-458b-4597-8da0-5fe17e7dc3d8" path="/var/lib/kubelet/pods/816fb468-458b-4597-8da0-5fe17e7dc3d8/volumes" Oct 30 01:21:29.786655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.786622 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc"] Oct 30 01:21:29.787422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787403 4927 cpu_manager.go:414] "containerMap: removing container" podUID="538ea5e7-d6f3-4391-97f6-4f7389bddb83" containerName="payload-pre-processing" Oct 30 01:21:29.787422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787419 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538ea5e7-d6f3-4391-97f6-4f7389bddb83" containerName="payload-pre-processing" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787435 4927 cpu_manager.go:414] "containerMap: removing container" podUID="79b8fa70-f329-4ae1-8384-068f7fd1d151" containerName="maas-api" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787441 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b8fa70-f329-4ae1-8384-068f7fd1d151" containerName="maas-api" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787453 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2677740b-3783-4ff1-8ff8-7c05c34a7ab5" containerName="payload-processing" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787458 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2677740b-3783-4ff1-8ff8-7c05c34a7ab5" containerName="payload-processing" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787468 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787473 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787481 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787485 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787496 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3dba0780-044e-415d-995e-3c02dd91ad75" containerName="maas-api" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787501 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dba0780-044e-415d-995e-3c02dd91ad75" containerName="maas-api" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787517 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787521 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787541 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c407565c-0ba4-4410-bedc-e6c9d4e5afe7" containerName="payload-pre-processing" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787546 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c407565c-0ba4-4410-bedc-e6c9d4e5afe7" containerName="payload-pre-processing" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787556 4927 cpu_manager.go:414] "containerMap: removing container" podUID="816fb468-458b-4597-8da0-5fe17e7dc3d8" containerName="payload-processing" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787561 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816fb468-458b-4597-8da0-5fe17e7dc3d8" containerName="payload-processing" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787581 4927 cpu_manager.go:414] "containerMap: removing container" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787588 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787606 4927 cpu_manager.go:414] "containerMap: removing container" podUID="07e63105-a131-4eb5-8055-e6dcc033c00b" containerName="payload-pre-processing" Oct 30 01:21:29.787642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787612 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e63105-a131-4eb5-8055-e6dcc033c00b" containerName="payload-pre-processing" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787716 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dba0780-044e-415d-995e-3c02dd91ad75" containerName="maas-api" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787728 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41bd266-1e41-448a-bf1c-ca43487dd741" containerName="istio-proxy" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787739 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c407565c-0ba4-4410-bedc-e6c9d4e5afe7" containerName="payload-pre-processing" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787746 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="816fb468-458b-4597-8da0-5fe17e7dc3d8" containerName="payload-processing" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787751 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="667e312d-afd1-4032-909f-aa766af53204" containerName="istio-proxy" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787759 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e63105-a131-4eb5-8055-e6dcc033c00b" containerName="payload-pre-processing" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787765 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="538ea5e7-d6f3-4391-97f6-4f7389bddb83" containerName="payload-pre-processing" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787772 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b8fa70-f329-4ae1-8384-068f7fd1d151" containerName="maas-api" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787777 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2677740b-3783-4ff1-8ff8-7c05c34a7ab5" containerName="payload-processing" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787783 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d9aaa7-e667-4fb5-86a7-b4a0389dd87d" containerName="istio-proxy" Oct 30 01:21:29.788746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.787788 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a43e4d1-4dad-4473-82e8-3c51b2bb8753" containerName="istio-proxy" Oct 30 01:21:29.793408 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.793387 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:29.816992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.816965 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc"] Oct 30 01:21:29.820047 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.820021 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-abf8ae\"/\"openshift-service-ca.crt\"" Oct 30 01:21:29.820180 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.820043 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-abf8ae\"/\"kube-root-ca.crt\"" Oct 30 01:21:29.820180 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.820116 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-abf8ae\"/\"default-dockercfg-k27zd\"" Oct 30 01:21:29.820316 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.820298 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-abf8ae\"/\"ipp-route-abf8ae-kserve-self-signed-certs\"" Oct 30 01:21:29.949290 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.949256 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-tmp-dir\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:29.949480 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.949317 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrbrv\" (UniqueName: \"kubernetes.io/projected/888a0308-4626-478a-80c7-efcdcb04cf19-kube-api-access-vrbrv\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:29.949546 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.949491 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-home\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:29.949691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.949674 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-model-cache\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:29.949730 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.949717 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-dshm\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:29.949840 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:29.949820 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888a0308-4626-478a-80c7-efcdcb04cf19-tls-certs\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.075596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.070850 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-home\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.075596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.071009 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-model-cache\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.075596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.071046 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-dshm\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.075596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.071090 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888a0308-4626-478a-80c7-efcdcb04cf19-tls-certs\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.075596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.071176 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-tmp-dir\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.075596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.071242 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrbrv\" (UniqueName: \"kubernetes.io/projected/888a0308-4626-478a-80c7-efcdcb04cf19-kube-api-access-vrbrv\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.075596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.072049 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-home\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.075596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.072302 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-model-cache\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.075596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.072915 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-tmp-dir\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.092203 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.092170 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-dshm\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.101634 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.099928 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888a0308-4626-478a-80c7-efcdcb04cf19-tls-certs\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.139471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.139396 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrbrv\" (UniqueName: \"kubernetes.io/projected/888a0308-4626-478a-80c7-efcdcb04cf19-kube-api-access-vrbrv\") pod \"ipp-route-abf8ae-kserve-55d6f9c697-h4hlc\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.419159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.419126 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:30.734762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:30.734733 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc"] Oct 30 01:21:30.735090 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:21:30.734733 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod888a0308_4626_478a_80c7_efcdcb04cf19.slice/crio-b84fbde7b43554d38b3bc17d18a15ba9584954b945a6a9fcaeda0be895612389 WatchSource:0}: Error finding container b84fbde7b43554d38b3bc17d18a15ba9584954b945a6a9fcaeda0be895612389: Status 404 returned error can't find the container with id b84fbde7b43554d38b3bc17d18a15ba9584954b945a6a9fcaeda0be895612389 Oct 30 01:21:31.551675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:31.551628 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" event={"ID":"888a0308-4626-478a-80c7-efcdcb04cf19","Type":"ContainerStarted","Data":"be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121"} Oct 30 01:21:31.551916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:31.551691 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" event={"ID":"888a0308-4626-478a-80c7-efcdcb04cf19","Type":"ContainerStarted","Data":"b84fbde7b43554d38b3bc17d18a15ba9584954b945a6a9fcaeda0be895612389"} Oct 30 01:21:31.594970 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:31.594918 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" podStartSLOduration=2.594905432 podStartE2EDuration="2.594905432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:31.591060558 +0000 UTC m=+2830.162986780" watchObservedRunningTime="2026-10-30 01:21:31.594905432 +0000 UTC m=+2830.166831653" Oct 30 01:21:32.420131 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:32.420100 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:34.445095 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:34.445057 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:21:36.109369 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.109329 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt"] Oct 30 01:21:36.109907 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.109613 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" podUID="6436b549-407a-473c-9374-23987295bbec" containerName="maas-api" containerID="cri-o://ceb845f283448f3fbe55d9d4875d1935534057eda2ea77dd0ab6f3d3e03079fd" gracePeriod=30 Oct 30 01:21:36.716642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.716604 4927 generic.go:356] "Generic (PLEG): container finished" podID="6436b549-407a-473c-9374-23987295bbec" containerID="ceb845f283448f3fbe55d9d4875d1935534057eda2ea77dd0ab6f3d3e03079fd" exitCode=0 Oct 30 01:21:36.717333 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.717000 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" event={"ID":"6436b549-407a-473c-9374-23987295bbec","Type":"ContainerDied","Data":"ceb845f283448f3fbe55d9d4875d1935534057eda2ea77dd0ab6f3d3e03079fd"} Oct 30 01:21:36.717534 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.717513 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" event={"ID":"6436b549-407a-473c-9374-23987295bbec","Type":"ContainerDied","Data":"c69c71c14b233ec6bfc665956b0057a92fa55718a2df2ee08c16300038fbfa06"} Oct 30 01:21:36.717693 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.717677 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c69c71c14b233ec6bfc665956b0057a92fa55718a2df2ee08c16300038fbfa06" Oct 30 01:21:36.718826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.718724 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 01:21:36.878605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.874230 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-metrics-certs\") pod \"6436b549-407a-473c-9374-23987295bbec\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " Oct 30 01:21:36.878605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.874314 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6436b549-407a-473c-9374-23987295bbec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6436b549-407a-473c-9374-23987295bbec-openshift-service-ca\") pod \"6436b549-407a-473c-9374-23987295bbec\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " Oct 30 01:21:36.878605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.874345 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6436b549-407a-473c-9374-23987295bbec-kube-api-access-6j86w\" (UniqueName: \"kubernetes.io/projected/6436b549-407a-473c-9374-23987295bbec-kube-api-access-6j86w\") pod \"6436b549-407a-473c-9374-23987295bbec\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " Oct 30 01:21:36.878605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.874601 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-maas-api-tls\") pod \"6436b549-407a-473c-9374-23987295bbec\" (UID: \"6436b549-407a-473c-9374-23987295bbec\") " Oct 30 01:21:36.881281 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.881245 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6436b549-407a-473c-9374-23987295bbec-openshift-service-ca" pod "6436b549-407a-473c-9374-23987295bbec" (UID: "6436b549-407a-473c-9374-23987295bbec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:36.883744 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.883715 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-metrics-certs" pod "6436b549-407a-473c-9374-23987295bbec" (UID: "6436b549-407a-473c-9374-23987295bbec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:36.884008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.883986 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-maas-api-tls" pod "6436b549-407a-473c-9374-23987295bbec" (UID: "6436b549-407a-473c-9374-23987295bbec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:36.884163 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:36.884143 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6436b549-407a-473c-9374-23987295bbec-kube-api-access-6j86w" pod "6436b549-407a-473c-9374-23987295bbec" (UID: "6436b549-407a-473c-9374-23987295bbec"). InnerVolumeSpecName "kube-api-access-6j86w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:37.001841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:37.001806 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.001841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:37.001842 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6436b549-407a-473c-9374-23987295bbec-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.002108 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:37.001858 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j86w\" (UniqueName: \"kubernetes.io/projected/6436b549-407a-473c-9374-23987295bbec-kube-api-access-6j86w\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.002108 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:37.001871 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6436b549-407a-473c-9374-23987295bbec-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.725840 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:37.725807 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt" Oct 30 01:21:37.783977 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:37.782695 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt"] Oct 30 01:21:37.793315 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:37.793279 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b3593a-769c5bf4d6-wdwlt"] Oct 30 01:21:39.573404 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:39.573367 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6436b549-407a-473c-9374-23987295bbec" path="/var/lib/kubelet/pods/6436b549-407a-473c-9374-23987295bbec/volumes" Oct 30 01:21:40.419581 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.419548 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:40.432151 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.432122 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:40.765326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.765243 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:21:40.924864 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.924829 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z"] Oct 30 01:21:40.926245 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.926202 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerName="istio-proxy" containerID="cri-o://13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411" gracePeriod=30 Oct 30 01:21:40.984897 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.984819 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q"] Oct 30 01:21:40.985185 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.985139 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" containerID="cri-o://3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1" gracePeriod=30 Oct 30 01:21:40.992883 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.992852 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx"] Oct 30 01:21:40.993136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.993107 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" containerID="cri-o://b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e" gracePeriod=30 Oct 30 01:21:40.999993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:40.999937 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t"] Oct 30 01:21:41.000225 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:41.000201 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" containerID="cri-o://59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43" gracePeriod=30 Oct 30 01:21:41.006829 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:41.006808 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn"] Oct 30 01:21:41.007033 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:41.007010 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" containerID="cri-o://8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746" gracePeriod=30 Oct 30 01:21:41.722877 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:41.722836 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:21:41.723077 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:41.722904 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:21:43.685173 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:43.685133 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn"] Oct 30 01:21:43.685668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:43.685388 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" containerID="cri-o://24925b422f3e7c08743e97771f69e3fdf0696c6f26132aadc40d7ddf6ee3a161" gracePeriod=30 Oct 30 01:21:44.028656 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.024820 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl"] Oct 30 01:21:44.028656 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.025137 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" podUID="d9941506-88cf-4d6e-8736-719ef1438fe6" containerName="maas-api" containerID="cri-o://52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4" gracePeriod=30 Oct 30 01:21:44.363255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.363224 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:21:44.430353 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.430315 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9941506-88cf-4d6e-8736-719ef1438fe6-kube-api-access-jmzhm\" (UniqueName: \"kubernetes.io/projected/d9941506-88cf-4d6e-8736-719ef1438fe6-kube-api-access-jmzhm\") pod \"d9941506-88cf-4d6e-8736-719ef1438fe6\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " Oct 30 01:21:44.430541 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.430478 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-maas-api-tls\") pod \"d9941506-88cf-4d6e-8736-719ef1438fe6\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " Oct 30 01:21:44.430644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.430624 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9941506-88cf-4d6e-8736-719ef1438fe6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9941506-88cf-4d6e-8736-719ef1438fe6-openshift-service-ca\") pod \"d9941506-88cf-4d6e-8736-719ef1438fe6\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " Oct 30 01:21:44.430762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.430744 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-metrics-certs\") pod \"d9941506-88cf-4d6e-8736-719ef1438fe6\" (UID: \"d9941506-88cf-4d6e-8736-719ef1438fe6\") " Oct 30 01:21:44.431038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.431011 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9941506-88cf-4d6e-8736-719ef1438fe6-openshift-service-ca" pod "d9941506-88cf-4d6e-8736-719ef1438fe6" (UID: "d9941506-88cf-4d6e-8736-719ef1438fe6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:44.431717 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.431688 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9941506-88cf-4d6e-8736-719ef1438fe6-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:44.434222 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.434195 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-metrics-certs" pod "d9941506-88cf-4d6e-8736-719ef1438fe6" (UID: "d9941506-88cf-4d6e-8736-719ef1438fe6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:44.434466 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.434448 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-maas-api-tls" pod "d9941506-88cf-4d6e-8736-719ef1438fe6" (UID: "d9941506-88cf-4d6e-8736-719ef1438fe6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:44.434660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.434633 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9941506-88cf-4d6e-8736-719ef1438fe6-kube-api-access-jmzhm" pod "d9941506-88cf-4d6e-8736-719ef1438fe6" (UID: "d9941506-88cf-4d6e-8736-719ef1438fe6"). InnerVolumeSpecName "kube-api-access-jmzhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:44.533192 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.533154 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmzhm\" (UniqueName: \"kubernetes.io/projected/d9941506-88cf-4d6e-8736-719ef1438fe6-kube-api-access-jmzhm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:44.533396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.533188 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:44.533396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.533227 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9941506-88cf-4d6e-8736-719ef1438fe6-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:44.808478 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.808440 4927 generic.go:356] "Generic (PLEG): container finished" podID="d9941506-88cf-4d6e-8736-719ef1438fe6" containerID="52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4" exitCode=0 Oct 30 01:21:44.809037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.808506 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" Oct 30 01:21:44.809037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.808721 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" event={"ID":"d9941506-88cf-4d6e-8736-719ef1438fe6","Type":"ContainerDied","Data":"52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4"} Oct 30 01:21:44.809037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.808775 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl" event={"ID":"d9941506-88cf-4d6e-8736-719ef1438fe6","Type":"ContainerDied","Data":"ea8d967668c91e1b2305a5c5cee2259465fce9ccb68b66cb3d3392e2c06e86c3"} Oct 30 01:21:44.809037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.808807 4927 scope.go:122] "RemoveContainer" containerID="52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4" Oct 30 01:21:44.820956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.820933 4927 scope.go:122] "RemoveContainer" containerID="52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4" Oct 30 01:21:44.821243 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:44.821215 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4\": container with ID starting with 52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4 not found: ID does not exist" containerID="52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4" Oct 30 01:21:44.821310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.821251 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4"} err="failed to get container status \"52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4\": rpc error: code = NotFound desc = could not find container \"52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4\": container with ID starting with 52cc251ea7ebe6ac1e2766bd68f689eb387f7d642a570f3fd67b50c484a3cbe4 not found: ID does not exist" Oct 30 01:21:44.863854 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.863822 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl"] Oct 30 01:21:44.875687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:44.875662 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-481249-5c56888b6f-fkdsl"] Oct 30 01:21:45.573423 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:45.573388 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9941506-88cf-4d6e-8736-719ef1438fe6" path="/var/lib/kubelet/pods/d9941506-88cf-4d6e-8736-719ef1438fe6/volumes" Oct 30 01:21:46.318630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.316532 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 01:21:46.338294 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.338255 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372083 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-istio-token\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-istio-token\") pod \"59b29146-fe62-4512-b201-b96133d2b510\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372129 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b29146-fe62-4512-b201-b96133d2b510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b29146-fe62-4512-b201-b96133d2b510-istiod-ca-cert\") pod \"59b29146-fe62-4512-b201-b96133d2b510\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372187 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-envoy\") pod \"ee87ca52-c410-4f3b-9906-ff823188c656\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372225 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-credential-socket\") pod \"59b29146-fe62-4512-b201-b96133d2b510\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372253 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-istio-token\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-istio-token\") pod \"ee87ca52-c410-4f3b-9906-ff823188c656\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372371 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-envoy\") pod \"59b29146-fe62-4512-b201-b96133d2b510\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372436 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-socket\") pod \"59b29146-fe62-4512-b201-b96133d2b510\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372496 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-data\") pod \"ee87ca52-c410-4f3b-9906-ff823188c656\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372532 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-certs\") pod \"ee87ca52-c410-4f3b-9906-ff823188c656\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372595 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee87ca52-c410-4f3b-9906-ff823188c656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee87ca52-c410-4f3b-9906-ff823188c656-istiod-ca-cert\") pod \"ee87ca52-c410-4f3b-9906-ff823188c656\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372665 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-data\") pod \"59b29146-fe62-4512-b201-b96133d2b510\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372720 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-credential-socket\") pod \"ee87ca52-c410-4f3b-9906-ff823188c656\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372777 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-socket\") pod \"ee87ca52-c410-4f3b-9906-ff823188c656\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372893 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-kube-api-access-4bc5q\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-kube-api-access-4bc5q\") pod \"ee87ca52-c410-4f3b-9906-ff823188c656\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.372961 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee87ca52-c410-4f3b-9906-ff823188c656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee87ca52-c410-4f3b-9906-ff823188c656-istio-podinfo\") pod \"ee87ca52-c410-4f3b-9906-ff823188c656\" (UID: \"ee87ca52-c410-4f3b-9906-ff823188c656\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.373004 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59b29146-fe62-4512-b201-b96133d2b510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b29146-fe62-4512-b201-b96133d2b510-istio-podinfo\") pod \"59b29146-fe62-4512-b201-b96133d2b510\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.373064 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-certs\") pod \"59b29146-fe62-4512-b201-b96133d2b510\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " Oct 30 01:21:46.374233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.373109 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-kube-api-access-5lv5t\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-kube-api-access-5lv5t\") pod \"59b29146-fe62-4512-b201-b96133d2b510\" (UID: \"59b29146-fe62-4512-b201-b96133d2b510\") " Oct 30 01:21:46.378262 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.378234 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-certs" pod "ee87ca52-c410-4f3b-9906-ff823188c656" (UID: "ee87ca52-c410-4f3b-9906-ff823188c656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.378912 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.378886 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b29146-fe62-4512-b201-b96133d2b510-istiod-ca-cert" pod "59b29146-fe62-4512-b201-b96133d2b510" (UID: "59b29146-fe62-4512-b201-b96133d2b510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:46.379340 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.379321 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-credential-socket" pod "59b29146-fe62-4512-b201-b96133d2b510" (UID: "59b29146-fe62-4512-b201-b96133d2b510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.381452 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.381423 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-kube-api-access-5lv5t" pod "59b29146-fe62-4512-b201-b96133d2b510" (UID: "59b29146-fe62-4512-b201-b96133d2b510"). InnerVolumeSpecName "kube-api-access-5lv5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.381687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.381666 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-socket" pod "ee87ca52-c410-4f3b-9906-ff823188c656" (UID: "ee87ca52-c410-4f3b-9906-ff823188c656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.382503 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.382478 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee87ca52-c410-4f3b-9906-ff823188c656-istiod-ca-cert" pod "ee87ca52-c410-4f3b-9906-ff823188c656" (UID: "ee87ca52-c410-4f3b-9906-ff823188c656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:46.382802 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.382777 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-data" pod "59b29146-fe62-4512-b201-b96133d2b510" (UID: "59b29146-fe62-4512-b201-b96133d2b510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.383091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.383064 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-credential-socket" pod "ee87ca52-c410-4f3b-9906-ff823188c656" (UID: "ee87ca52-c410-4f3b-9906-ff823188c656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.383312 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.383288 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-socket" pod "59b29146-fe62-4512-b201-b96133d2b510" (UID: "59b29146-fe62-4512-b201-b96133d2b510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.404751 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.404530 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-certs" pod "59b29146-fe62-4512-b201-b96133d2b510" (UID: "59b29146-fe62-4512-b201-b96133d2b510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.405011 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.404922 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-data" pod "ee87ca52-c410-4f3b-9906-ff823188c656" (UID: "ee87ca52-c410-4f3b-9906-ff823188c656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.407909 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.407861 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee87ca52-c410-4f3b-9906-ff823188c656-istio-podinfo" pod "ee87ca52-c410-4f3b-9906-ff823188c656" (UID: "ee87ca52-c410-4f3b-9906-ff823188c656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:46.408057 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.407912 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-envoy" pod "59b29146-fe62-4512-b201-b96133d2b510" (UID: "59b29146-fe62-4512-b201-b96133d2b510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.409697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.409659 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-istio-token" pod "59b29146-fe62-4512-b201-b96133d2b510" (UID: "59b29146-fe62-4512-b201-b96133d2b510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.409900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.409868 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b29146-fe62-4512-b201-b96133d2b510-istio-podinfo" pod "59b29146-fe62-4512-b201-b96133d2b510" (UID: "59b29146-fe62-4512-b201-b96133d2b510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:46.410160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.410127 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-envoy" pod "ee87ca52-c410-4f3b-9906-ff823188c656" (UID: "ee87ca52-c410-4f3b-9906-ff823188c656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.411845 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.411821 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-istio-token" pod "ee87ca52-c410-4f3b-9906-ff823188c656" (UID: "ee87ca52-c410-4f3b-9906-ff823188c656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.412386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.412364 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-kube-api-access-4bc5q" pod "ee87ca52-c410-4f3b-9906-ff823188c656" (UID: "ee87ca52-c410-4f3b-9906-ff823188c656"). InnerVolumeSpecName "kube-api-access-4bc5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.434550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.434523 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:21:46.437738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.437716 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:21:46.442355 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.442331 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:21:46.475621 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475520 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-credential-socket\") pod \"457b5a9b-2a51-43fe-9104-f7a09680fa16\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " Oct 30 01:21:46.475785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475625 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-podinfo\") pod \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " Oct 30 01:21:46.475785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475671 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-data\") pod \"dae00aad-022f-4c43-9680-1dd55a8cad48\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " Oct 30 01:21:46.475785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475710 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-socket\") pod \"dae00aad-022f-4c43-9680-1dd55a8cad48\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " Oct 30 01:21:46.475785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475739 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-certs\") pod \"457b5a9b-2a51-43fe-9104-f7a09680fa16\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " Oct 30 01:21:46.476027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475807 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-token\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-token\") pod \"457b5a9b-2a51-43fe-9104-f7a09680fa16\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " Oct 30 01:21:46.476027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475836 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae00aad-022f-4c43-9680-1dd55a8cad48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae00aad-022f-4c43-9680-1dd55a8cad48-istiod-ca-cert\") pod \"dae00aad-022f-4c43-9680-1dd55a8cad48\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " Oct 30 01:21:46.476027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475865 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-kube-api-access-kjmrm\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-kube-api-access-kjmrm\") pod \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " Oct 30 01:21:46.476027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475967 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-kube-api-access-bddlj\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-kube-api-access-bddlj\") pod \"dae00aad-022f-4c43-9680-1dd55a8cad48\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " Oct 30 01:21:46.476027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.475995 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istiod-ca-cert\") pod \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " Oct 30 01:21:46.476300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476033 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-data\") pod \"457b5a9b-2a51-43fe-9104-f7a09680fa16\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " Oct 30 01:21:46.476300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476081 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-socket\") pod \"457b5a9b-2a51-43fe-9104-f7a09680fa16\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " Oct 30 01:21:46.476300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476117 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-certs\") pod \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " Oct 30 01:21:46.476300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476153 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-podinfo\") pod \"457b5a9b-2a51-43fe-9104-f7a09680fa16\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " Oct 30 01:21:46.476300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476198 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-credential-socket\") pod \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " Oct 30 01:21:46.476300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476261 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-data\") pod \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " Oct 30 01:21:46.476646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476301 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-envoy\") pod \"dae00aad-022f-4c43-9680-1dd55a8cad48\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " Oct 30 01:21:46.476646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476359 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-socket\") pod \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " Oct 30 01:21:46.476646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476391 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-envoy\") pod \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " Oct 30 01:21:46.476646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476428 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-envoy\") pod \"457b5a9b-2a51-43fe-9104-f7a09680fa16\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " Oct 30 01:21:46.476646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476466 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-certs\") pod \"dae00aad-022f-4c43-9680-1dd55a8cad48\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " Oct 30 01:21:46.476646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476527 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-podinfo\") pod \"dae00aad-022f-4c43-9680-1dd55a8cad48\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " Oct 30 01:21:46.476646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476561 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-kube-api-access-2zj94\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-kube-api-access-2zj94\") pod \"457b5a9b-2a51-43fe-9104-f7a09680fa16\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " Oct 30 01:21:46.476646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476628 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-token\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-token\") pod \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\" (UID: \"0a221756-2b9e-4fbc-a2ba-84b76f88fe89\") " Oct 30 01:21:46.477070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476668 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-token\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-token\") pod \"dae00aad-022f-4c43-9680-1dd55a8cad48\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " Oct 30 01:21:46.477070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476723 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-credential-socket\") pod \"dae00aad-022f-4c43-9680-1dd55a8cad48\" (UID: \"dae00aad-022f-4c43-9680-1dd55a8cad48\") " Oct 30 01:21:46.477070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.476753 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457b5a9b-2a51-43fe-9104-f7a09680fa16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b5a9b-2a51-43fe-9104-f7a09680fa16-istiod-ca-cert\") pod \"457b5a9b-2a51-43fe-9104-f7a09680fa16\" (UID: \"457b5a9b-2a51-43fe-9104-f7a09680fa16\") " Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477679 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bc5q\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-kube-api-access-4bc5q\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477711 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee87ca52-c410-4f3b-9906-ff823188c656-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477727 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b29146-fe62-4512-b201-b96133d2b510-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477742 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477756 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lv5t\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-kube-api-access-5lv5t\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477770 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b29146-fe62-4512-b201-b96133d2b510-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477786 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b29146-fe62-4512-b201-b96133d2b510-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477801 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477813 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477827 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee87ca52-c410-4f3b-9906-ff823188c656-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477839 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477852 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477865 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477879 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477892 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee87ca52-c410-4f3b-9906-ff823188c656-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477905 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b29146-fe62-4512-b201-b96133d2b510-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477918 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.477932 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee87ca52-c410-4f3b-9906-ff823188c656-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.478315 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457b5a9b-2a51-43fe-9104-f7a09680fa16-istiod-ca-cert" pod "457b5a9b-2a51-43fe-9104-f7a09680fa16" (UID: "457b5a9b-2a51-43fe-9104-f7a09680fa16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.478658 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-certs" pod "dae00aad-022f-4c43-9680-1dd55a8cad48" (UID: "dae00aad-022f-4c43-9680-1dd55a8cad48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.478968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.478668 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-data" pod "0a221756-2b9e-4fbc-a2ba-84b76f88fe89" (UID: "0a221756-2b9e-4fbc-a2ba-84b76f88fe89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.480532 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.480268 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-certs" pod "0a221756-2b9e-4fbc-a2ba-84b76f88fe89" (UID: "0a221756-2b9e-4fbc-a2ba-84b76f88fe89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.480984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.480775 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istiod-ca-cert" pod "0a221756-2b9e-4fbc-a2ba-84b76f88fe89" (UID: "0a221756-2b9e-4fbc-a2ba-84b76f88fe89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:46.481269 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.481117 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-data" pod "457b5a9b-2a51-43fe-9104-f7a09680fa16" (UID: "457b5a9b-2a51-43fe-9104-f7a09680fa16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.481699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.481417 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-socket" pod "457b5a9b-2a51-43fe-9104-f7a09680fa16" (UID: "457b5a9b-2a51-43fe-9104-f7a09680fa16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.482261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.482011 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-credential-socket" pod "0a221756-2b9e-4fbc-a2ba-84b76f88fe89" (UID: "0a221756-2b9e-4fbc-a2ba-84b76f88fe89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.483631 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.483603 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-credential-socket" pod "dae00aad-022f-4c43-9680-1dd55a8cad48" (UID: "dae00aad-022f-4c43-9680-1dd55a8cad48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.483915 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.483891 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-socket" pod "0a221756-2b9e-4fbc-a2ba-84b76f88fe89" (UID: "0a221756-2b9e-4fbc-a2ba-84b76f88fe89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.486682 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.486196 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-data" pod "dae00aad-022f-4c43-9680-1dd55a8cad48" (UID: "dae00aad-022f-4c43-9680-1dd55a8cad48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.486682 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.486382 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-credential-socket" pod "457b5a9b-2a51-43fe-9104-f7a09680fa16" (UID: "457b5a9b-2a51-43fe-9104-f7a09680fa16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.490918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.489872 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-podinfo" pod "dae00aad-022f-4c43-9680-1dd55a8cad48" (UID: "dae00aad-022f-4c43-9680-1dd55a8cad48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:46.492368 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.492117 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae00aad-022f-4c43-9680-1dd55a8cad48-istiod-ca-cert" pod "dae00aad-022f-4c43-9680-1dd55a8cad48" (UID: "dae00aad-022f-4c43-9680-1dd55a8cad48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:46.492772 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.492597 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-socket" pod "dae00aad-022f-4c43-9680-1dd55a8cad48" (UID: "dae00aad-022f-4c43-9680-1dd55a8cad48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.493188 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.492917 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-certs" pod "457b5a9b-2a51-43fe-9104-f7a09680fa16" (UID: "457b5a9b-2a51-43fe-9104-f7a09680fa16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.498102 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.498076 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-token" pod "0a221756-2b9e-4fbc-a2ba-84b76f88fe89" (UID: "0a221756-2b9e-4fbc-a2ba-84b76f88fe89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.498326 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.498306 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-token" pod "dae00aad-022f-4c43-9680-1dd55a8cad48" (UID: "dae00aad-022f-4c43-9680-1dd55a8cad48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.501179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.501075 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-kube-api-access-bddlj" pod "dae00aad-022f-4c43-9680-1dd55a8cad48" (UID: "dae00aad-022f-4c43-9680-1dd55a8cad48"). InnerVolumeSpecName "kube-api-access-bddlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.501179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.501154 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-kube-api-access-2zj94" pod "457b5a9b-2a51-43fe-9104-f7a09680fa16" (UID: "457b5a9b-2a51-43fe-9104-f7a09680fa16"). InnerVolumeSpecName "kube-api-access-2zj94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.501471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.501439 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-envoy" pod "457b5a9b-2a51-43fe-9104-f7a09680fa16" (UID: "457b5a9b-2a51-43fe-9104-f7a09680fa16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.504293 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.502472 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-kube-api-access-kjmrm" pod "0a221756-2b9e-4fbc-a2ba-84b76f88fe89" (UID: "0a221756-2b9e-4fbc-a2ba-84b76f88fe89"). InnerVolumeSpecName "kube-api-access-kjmrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.504293 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.502955 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-podinfo" pod "0a221756-2b9e-4fbc-a2ba-84b76f88fe89" (UID: "0a221756-2b9e-4fbc-a2ba-84b76f88fe89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:46.504670 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.504643 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-token" pod "457b5a9b-2a51-43fe-9104-f7a09680fa16" (UID: "457b5a9b-2a51-43fe-9104-f7a09680fa16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.504862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.504840 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-envoy" pod "0a221756-2b9e-4fbc-a2ba-84b76f88fe89" (UID: "0a221756-2b9e-4fbc-a2ba-84b76f88fe89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.504987 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.504967 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-podinfo" pod "457b5a9b-2a51-43fe-9104-f7a09680fa16" (UID: "457b5a9b-2a51-43fe-9104-f7a09680fa16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:46.507347 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.507314 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-envoy" pod "dae00aad-022f-4c43-9680-1dd55a8cad48" (UID: "dae00aad-022f-4c43-9680-1dd55a8cad48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:46.579785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579739 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bddlj\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-kube-api-access-bddlj\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579778 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579795 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579810 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579824 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579837 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579851 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579864 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579878 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579890 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579902 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579914 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579927 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579940 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579954 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zj94\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-kube-api-access-2zj94\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579967 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579979 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.579992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.579993 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.580550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.580007 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b5a9b-2a51-43fe-9104-f7a09680fa16-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.580550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.580022 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.580550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.580035 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.580550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.580047 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.580550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.580060 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae00aad-022f-4c43-9680-1dd55a8cad48-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.580550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.580073 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b5a9b-2a51-43fe-9104-f7a09680fa16-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.580550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.580086 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457b5a9b-2a51-43fe-9104-f7a09680fa16-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.580550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.580098 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae00aad-022f-4c43-9680-1dd55a8cad48-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.580550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.580110 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjmrm\" (UniqueName: \"kubernetes.io/projected/0a221756-2b9e-4fbc-a2ba-84b76f88fe89-kube-api-access-kjmrm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.827813 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.827734 4927 generic.go:356] "Generic (PLEG): container finished" podID="ee87ca52-c410-4f3b-9906-ff823188c656" containerID="3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1" exitCode=0 Oct 30 01:21:46.827813 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.827784 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" Oct 30 01:21:46.828031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.827816 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" event={"ID":"ee87ca52-c410-4f3b-9906-ff823188c656","Type":"ContainerDied","Data":"3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1"} Oct 30 01:21:46.828031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.827870 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q" event={"ID":"ee87ca52-c410-4f3b-9906-ff823188c656","Type":"ContainerDied","Data":"34826da47928c1e2d70d698da97f5fdc89ca54969f2893fc23fad8526e32c77a"} Oct 30 01:21:46.828031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.827903 4927 scope.go:122] "RemoveContainer" containerID="3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1" Oct 30 01:21:46.829471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.829446 4927 generic.go:356] "Generic (PLEG): container finished" podID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerID="13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411" exitCode=0 Oct 30 01:21:46.829620 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.829505 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" Oct 30 01:21:46.829696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.829556 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" event={"ID":"457b5a9b-2a51-43fe-9104-f7a09680fa16","Type":"ContainerDied","Data":"13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411"} Oct 30 01:21:46.829756 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.829711 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z" event={"ID":"457b5a9b-2a51-43fe-9104-f7a09680fa16","Type":"ContainerDied","Data":"e822740c1561401f0e1f0aae585107209b5a2ed1a3fdb933b7a5aca50d53b2ba"} Oct 30 01:21:46.831075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.831053 4927 generic.go:356] "Generic (PLEG): container finished" podID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerID="b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e" exitCode=0 Oct 30 01:21:46.831249 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.831092 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" event={"ID":"0a221756-2b9e-4fbc-a2ba-84b76f88fe89","Type":"ContainerDied","Data":"b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e"} Oct 30 01:21:46.831249 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.831151 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" event={"ID":"0a221756-2b9e-4fbc-a2ba-84b76f88fe89","Type":"ContainerDied","Data":"beebd076ae5e76922cffc4c366bad63f771b305ce9aa9db05b7b21edbc908eee"} Oct 30 01:21:46.831249 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.831107 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx" Oct 30 01:21:46.833267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.833089 4927 generic.go:356] "Generic (PLEG): container finished" podID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerID="8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746" exitCode=0 Oct 30 01:21:46.833267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.833150 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" Oct 30 01:21:46.833267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.833200 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" event={"ID":"dae00aad-022f-4c43-9680-1dd55a8cad48","Type":"ContainerDied","Data":"8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746"} Oct 30 01:21:46.833267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.833234 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn" event={"ID":"dae00aad-022f-4c43-9680-1dd55a8cad48","Type":"ContainerDied","Data":"f4bf8bc6c9198497f5bcfc11bccf73348489bdd35ee6a2995a6a9fc1d1e1b6be"} Oct 30 01:21:46.847429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.847402 4927 generic.go:356] "Generic (PLEG): container finished" podID="59b29146-fe62-4512-b201-b96133d2b510" containerID="59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43" exitCode=0 Oct 30 01:21:46.847544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.847442 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" event={"ID":"59b29146-fe62-4512-b201-b96133d2b510","Type":"ContainerDied","Data":"59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43"} Oct 30 01:21:46.847544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.847478 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" Oct 30 01:21:46.847544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.847480 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t" event={"ID":"59b29146-fe62-4512-b201-b96133d2b510","Type":"ContainerDied","Data":"42353f893db89bc149cf00f2874a926a92a7a31ff3a4406c5b3d7b56ddaac0b4"} Oct 30 01:21:46.851071 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.851053 4927 scope.go:122] "RemoveContainer" containerID="836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d" Oct 30 01:21:46.863451 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.863432 4927 scope.go:122] "RemoveContainer" containerID="3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1" Oct 30 01:21:46.863767 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:46.863747 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1\": container with ID starting with 3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1 not found: ID does not exist" containerID="3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1" Oct 30 01:21:46.863851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.863776 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1"} err="failed to get container status \"3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1\": rpc error: code = NotFound desc = could not find container \"3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1\": container with ID starting with 3608b780466623d538a7f6edb32945988a8597084904bc65428f0d80487c19b1 not found: ID does not exist" Oct 30 01:21:46.863851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.863795 4927 scope.go:122] "RemoveContainer" containerID="836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d" Oct 30 01:21:46.864030 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:46.864012 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d\": container with ID starting with 836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d not found: ID does not exist" containerID="836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d" Oct 30 01:21:46.864075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.864037 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d"} err="failed to get container status \"836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d\": rpc error: code = NotFound desc = could not find container \"836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d\": container with ID starting with 836a7515cc594ac0103f9542d58640b66c8fc44d73f95092bdf637299826080d not found: ID does not exist" Oct 30 01:21:46.864075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.864053 4927 scope.go:122] "RemoveContainer" containerID="13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411" Oct 30 01:21:46.875872 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.875849 4927 scope.go:122] "RemoveContainer" containerID="13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411" Oct 30 01:21:46.876161 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:46.876134 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411\": container with ID starting with 13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411 not found: ID does not exist" containerID="13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411" Oct 30 01:21:46.876253 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.876176 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411"} err="failed to get container status \"13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411\": rpc error: code = NotFound desc = could not find container \"13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411\": container with ID starting with 13585bc68d343f58ec9aa1f1c55f2a22e083a20aebfb0a92d4930e86aec9d411 not found: ID does not exist" Oct 30 01:21:46.876253 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.876202 4927 scope.go:122] "RemoveContainer" containerID="b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e" Oct 30 01:21:46.886203 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.886179 4927 scope.go:122] "RemoveContainer" containerID="b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e" Oct 30 01:21:46.886483 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:46.886464 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e\": container with ID starting with b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e not found: ID does not exist" containerID="b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e" Oct 30 01:21:46.886544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.886489 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e"} err="failed to get container status \"b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e\": rpc error: code = NotFound desc = could not find container \"b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e\": container with ID starting with b0f9a90daf7af855e28b0d1770ad02571424dcab0bc4e0302d5139596959b35e not found: ID does not exist" Oct 30 01:21:46.886544 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.886506 4927 scope.go:122] "RemoveContainer" containerID="8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746" Oct 30 01:21:46.900102 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.899915 4927 scope.go:122] "RemoveContainer" containerID="8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746" Oct 30 01:21:46.901452 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:46.900358 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746\": container with ID starting with 8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746 not found: ID does not exist" containerID="8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746" Oct 30 01:21:46.901452 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.900392 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746"} err="failed to get container status \"8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746\": rpc error: code = NotFound desc = could not find container \"8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746\": container with ID starting with 8904b3dc41555cad50df301e59cdfaeda44c63030170d67631d0da89225c3746 not found: ID does not exist" Oct 30 01:21:46.901452 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:46.900413 4927 scope.go:122] "RemoveContainer" containerID="59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43" Oct 30 01:21:47.069534 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.052814 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t"] Oct 30 01:21:47.069534 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.052877 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w8t8t"] Oct 30 01:21:47.074483 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.073356 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx"] Oct 30 01:21:47.104786 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.104751 4927 scope.go:122] "RemoveContainer" containerID="59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43" Oct 30 01:21:47.107555 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:21:47.107526 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43\": container with ID starting with 59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43 not found: ID does not exist" containerID="59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43" Oct 30 01:21:47.107891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.107837 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43"} err="failed to get container status \"59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43\": rpc error: code = NotFound desc = could not find container \"59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43\": container with ID starting with 59daef11e54bf5ddfc35f005a469bb5ff76fd93f5503c24dca364e1e5ca64f43 not found: ID does not exist" Oct 30 01:21:47.125669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.125637 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-2zbhx"] Oct 30 01:21:47.131902 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.131872 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q"] Oct 30 01:21:47.137018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.136998 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-ndf5q"] Oct 30 01:21:47.141924 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.141901 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z"] Oct 30 01:21:47.145699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.145677 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-kvx4z"] Oct 30 01:21:47.174524 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.174500 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn"] Oct 30 01:21:47.197763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.197690 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-8dszn"] Oct 30 01:21:47.573771 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.573737 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" path="/var/lib/kubelet/pods/0a221756-2b9e-4fbc-a2ba-84b76f88fe89/volumes" Oct 30 01:21:47.574377 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.574358 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" path="/var/lib/kubelet/pods/457b5a9b-2a51-43fe-9104-f7a09680fa16/volumes" Oct 30 01:21:47.574907 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.574892 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b29146-fe62-4512-b201-b96133d2b510" path="/var/lib/kubelet/pods/59b29146-fe62-4512-b201-b96133d2b510/volumes" Oct 30 01:21:47.575822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.575798 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" path="/var/lib/kubelet/pods/dae00aad-022f-4c43-9680-1dd55a8cad48/volumes" Oct 30 01:21:47.576290 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:47.576276 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" path="/var/lib/kubelet/pods/ee87ca52-c410-4f3b-9906-ff823188c656/volumes" Oct 30 01:21:48.895171 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:48.895134 4927 generic.go:356] "Generic (PLEG): container finished" podID="e117e98e-358b-422a-a2ab-0e80be395b68" containerID="24925b422f3e7c08743e97771f69e3fdf0696c6f26132aadc40d7ddf6ee3a161" exitCode=0 Oct 30 01:21:48.895606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:48.895409 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" event={"ID":"e117e98e-358b-422a-a2ab-0e80be395b68","Type":"ContainerDied","Data":"24925b422f3e7c08743e97771f69e3fdf0696c6f26132aadc40d7ddf6ee3a161"} Oct 30 01:21:49.006061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.006035 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.109790 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-socket\") pod \"e117e98e-358b-422a-a2ab-0e80be395b68\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.109871 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-data\") pod \"e117e98e-358b-422a-a2ab-0e80be395b68\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.109913 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-istio-token\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-istio-token\") pod \"e117e98e-358b-422a-a2ab-0e80be395b68\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.109987 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e117e98e-358b-422a-a2ab-0e80be395b68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e117e98e-358b-422a-a2ab-0e80be395b68-istiod-ca-cert\") pod \"e117e98e-358b-422a-a2ab-0e80be395b68\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.110013 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-envoy\") pod \"e117e98e-358b-422a-a2ab-0e80be395b68\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.110134 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-credential-socket\") pod \"e117e98e-358b-422a-a2ab-0e80be395b68\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.110180 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e117e98e-358b-422a-a2ab-0e80be395b68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e117e98e-358b-422a-a2ab-0e80be395b68-istio-podinfo\") pod \"e117e98e-358b-422a-a2ab-0e80be395b68\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.110261 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-kube-api-access-5bz8f\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-kube-api-access-5bz8f\") pod \"e117e98e-358b-422a-a2ab-0e80be395b68\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.110386 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-certs\") pod \"e117e98e-358b-422a-a2ab-0e80be395b68\" (UID: \"e117e98e-358b-422a-a2ab-0e80be395b68\") " Oct 30 01:21:49.112028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.111675 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e117e98e-358b-422a-a2ab-0e80be395b68-istiod-ca-cert" pod "e117e98e-358b-422a-a2ab-0e80be395b68" (UID: "e117e98e-358b-422a-a2ab-0e80be395b68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:49.149147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.113025 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-socket" pod "e117e98e-358b-422a-a2ab-0e80be395b68" (UID: "e117e98e-358b-422a-a2ab-0e80be395b68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.149147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.113239 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-data" pod "e117e98e-358b-422a-a2ab-0e80be395b68" (UID: "e117e98e-358b-422a-a2ab-0e80be395b68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.149147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.113615 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-certs" pod "e117e98e-358b-422a-a2ab-0e80be395b68" (UID: "e117e98e-358b-422a-a2ab-0e80be395b68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.149147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.114421 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-credential-socket" pod "e117e98e-358b-422a-a2ab-0e80be395b68" (UID: "e117e98e-358b-422a-a2ab-0e80be395b68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.169729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.166476 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-kube-api-access-5bz8f" pod "e117e98e-358b-422a-a2ab-0e80be395b68" (UID: "e117e98e-358b-422a-a2ab-0e80be395b68"). InnerVolumeSpecName "kube-api-access-5bz8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:49.169729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.168044 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e117e98e-358b-422a-a2ab-0e80be395b68-istio-podinfo" pod "e117e98e-358b-422a-a2ab-0e80be395b68" (UID: "e117e98e-358b-422a-a2ab-0e80be395b68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:49.170686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.170632 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-istio-token" pod "e117e98e-358b-422a-a2ab-0e80be395b68" (UID: "e117e98e-358b-422a-a2ab-0e80be395b68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:49.197492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.196709 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-envoy" pod "e117e98e-358b-422a-a2ab-0e80be395b68" (UID: "e117e98e-358b-422a-a2ab-0e80be395b68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:49.212806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.212770 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e117e98e-358b-422a-a2ab-0e80be395b68-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.212806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.212809 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.213012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.212827 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.213012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.212840 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e117e98e-358b-422a-a2ab-0e80be395b68-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.213012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.212857 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bz8f\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-kube-api-access-5bz8f\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.213012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.212869 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.213012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.212883 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.213012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.212896 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e117e98e-358b-422a-a2ab-0e80be395b68-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.213012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.212941 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e117e98e-358b-422a-a2ab-0e80be395b68-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.494890 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8"] Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496016 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496039 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496060 4927 cpu_manager.go:414] "containerMap: removing container" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496069 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496091 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496101 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496112 4927 cpu_manager.go:414] "containerMap: removing container" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496121 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496140 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d9941506-88cf-4d6e-8736-719ef1438fe6" containerName="maas-api" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496147 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9941506-88cf-4d6e-8736-719ef1438fe6" containerName="maas-api" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496165 4927 cpu_manager.go:414] "containerMap: removing container" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496173 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496188 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496194 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496229 4927 cpu_manager.go:414] "containerMap: removing container" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496238 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496266 4927 cpu_manager.go:414] "containerMap: removing container" podUID="6436b549-407a-473c-9374-23987295bbec" containerName="maas-api" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496274 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6436b549-407a-473c-9374-23987295bbec" containerName="maas-api" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496435 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b29146-fe62-4512-b201-b96133d2b510" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496451 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="6436b549-407a-473c-9374-23987295bbec" containerName="maas-api" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496463 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496473 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="457b5a9b-2a51-43fe-9104-f7a09680fa16" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496483 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a221756-2b9e-4fbc-a2ba-84b76f88fe89" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496492 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9941506-88cf-4d6e-8736-719ef1438fe6" containerName="maas-api" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496505 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae00aad-022f-4c43-9680-1dd55a8cad48" containerName="istio-proxy" Oct 30 01:21:49.498594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.496516 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" containerName="istio-proxy" Oct 30 01:21:49.506000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.505975 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:21:49.520687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.520545 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb426a-d010-427a-a153-569fee89d6a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8\" (UID: \"82cb426a-d010-427a-a153-569fee89d6a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:21:49.520687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.520654 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566rd\" (UniqueName: \"kubernetes.io/projected/82cb426a-d010-427a-a153-569fee89d6a7-kube-api-access-566rd\") pod \"payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8\" (UID: \"82cb426a-d010-427a-a153-569fee89d6a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:21:49.521900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.521338 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5f99e7\"" Oct 30 01:21:49.525859 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.525831 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8"] Oct 30 01:21:49.556062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.556008 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5f99e7-dockercfg-gnrck\"" Oct 30 01:21:49.622853 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.622821 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb426a-d010-427a-a153-569fee89d6a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8\" (UID: \"82cb426a-d010-427a-a153-569fee89d6a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:21:49.622991 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.622876 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566rd\" (UniqueName: \"kubernetes.io/projected/82cb426a-d010-427a-a153-569fee89d6a7-kube-api-access-566rd\") pod \"payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8\" (UID: \"82cb426a-d010-427a-a153-569fee89d6a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:21:49.623639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.623616 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb426a-d010-427a-a153-569fee89d6a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8\" (UID: \"82cb426a-d010-427a-a153-569fee89d6a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:21:49.653114 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.653084 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566rd\" (UniqueName: \"kubernetes.io/projected/82cb426a-d010-427a-a153-569fee89d6a7-kube-api-access-566rd\") pod \"payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8\" (UID: \"82cb426a-d010-427a-a153-569fee89d6a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:21:49.721549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.721516 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv"] Oct 30 01:21:49.722496 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.722471 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee87ca52-c410-4f3b-9906-ff823188c656" containerName="istio-proxy" Oct 30 01:21:49.726729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.726707 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:21:49.747338 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.747284 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv"] Oct 30 01:21:49.825269 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.825167 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbf8h\" (UniqueName: \"kubernetes.io/projected/1cd85074-3f4b-4d36-ad24-fa7abc24a367-kube-api-access-dbf8h\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv\" (UID: \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:21:49.825449 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.825324 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd85074-3f4b-4d36-ad24-fa7abc24a367-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv\" (UID: \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:21:49.830361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.830322 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:21:49.908906 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.908876 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" event={"ID":"e117e98e-358b-422a-a2ab-0e80be395b68","Type":"ContainerDied","Data":"18c7b9b9689e5d70b9ef74183dd9780d67902325c49f9bd4790b80d307c5585c"} Oct 30 01:21:49.909525 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.908946 4927 scope.go:122] "RemoveContainer" containerID="24925b422f3e7c08743e97771f69e3fdf0696c6f26132aadc40d7ddf6ee3a161" Oct 30 01:21:49.909525 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.909088 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn" Oct 30 01:21:49.928072 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.928038 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbf8h\" (UniqueName: \"kubernetes.io/projected/1cd85074-3f4b-4d36-ad24-fa7abc24a367-kube-api-access-dbf8h\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv\" (UID: \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:21:49.928186 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.928110 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd85074-3f4b-4d36-ad24-fa7abc24a367-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv\" (UID: \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:21:49.928864 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.928843 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd85074-3f4b-4d36-ad24-fa7abc24a367-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv\" (UID: \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:21:49.959121 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.959092 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbf8h\" (UniqueName: \"kubernetes.io/projected/1cd85074-3f4b-4d36-ad24-fa7abc24a367-kube-api-access-dbf8h\") pod \"payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv\" (UID: \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:21:49.978862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.978831 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn"] Oct 30 01:21:49.986955 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:49.986919 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-kmmmn"] Oct 30 01:21:50.045638 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.045603 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:21:50.144258 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.144225 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8"] Oct 30 01:21:50.145759 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:21:50.145715 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cb426a_d010_427a_a153_569fee89d6a7.slice/crio-150ad75b1cc12bdab382683b2680cc544f87cca978af9009f97e6be93a83885c WatchSource:0}: Error finding container 150ad75b1cc12bdab382683b2680cc544f87cca978af9009f97e6be93a83885c: Status 404 returned error can't find the container with id 150ad75b1cc12bdab382683b2680cc544f87cca978af9009f97e6be93a83885c Oct 30 01:21:50.477906 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.477862 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv"] Oct 30 01:21:50.515127 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:21:50.514482 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd85074_3f4b_4d36_ad24_fa7abc24a367.slice/crio-8e87fdfa8abd52cdd663857e1e44e11671f0b0895427e4564adf8e17998bed83 WatchSource:0}: Error finding container 8e87fdfa8abd52cdd663857e1e44e11671f0b0895427e4564adf8e17998bed83: Status 404 returned error can't find the container with id 8e87fdfa8abd52cdd663857e1e44e11671f0b0895427e4564adf8e17998bed83 Oct 30 01:21:50.918765 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.918698 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" event={"ID":"82cb426a-d010-427a-a153-569fee89d6a7","Type":"ContainerStarted","Data":"6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673"} Oct 30 01:21:50.919251 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.918827 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:21:50.919251 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.918852 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" event={"ID":"82cb426a-d010-427a-a153-569fee89d6a7","Type":"ContainerStarted","Data":"150ad75b1cc12bdab382683b2680cc544f87cca978af9009f97e6be93a83885c"} Oct 30 01:21:50.921683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.921649 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" event={"ID":"1cd85074-3f4b-4d36-ad24-fa7abc24a367","Type":"ContainerStarted","Data":"c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04"} Oct 30 01:21:50.921818 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.921699 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" event={"ID":"1cd85074-3f4b-4d36-ad24-fa7abc24a367","Type":"ContainerStarted","Data":"8e87fdfa8abd52cdd663857e1e44e11671f0b0895427e4564adf8e17998bed83"} Oct 30 01:21:50.921818 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.921759 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:21:50.959584 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.959517 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" podStartSLOduration=1.959503268 podStartE2EDuration="1.959503268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:50.958431714 +0000 UTC m=+2849.530357936" watchObservedRunningTime="2026-10-30 01:21:50.959503268 +0000 UTC m=+2849.531429490" Oct 30 01:21:50.999900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:50.999847 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" podStartSLOduration=1.999833586 podStartE2EDuration="1.999833586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:50.997825021 +0000 UTC m=+2849.569751244" watchObservedRunningTime="2026-10-30 01:21:50.999833586 +0000 UTC m=+2849.571759807" Oct 30 01:21:51.573699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:51.573665 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e117e98e-358b-422a-a2ab-0e80be395b68" path="/var/lib/kubelet/pods/e117e98e-358b-422a-a2ab-0e80be395b68/volumes" Oct 30 01:21:55.109786 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:55.109719 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm"] Oct 30 01:21:55.110245 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:55.110030 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" containerID="cri-o://fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8" gracePeriod=30 Oct 30 01:21:58.835697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:58.824449 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv"] Oct 30 01:21:58.835697 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:21:58.824728 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" containerID="cri-o://179ee5373677eff984a1961d70f2ae2d9355ea8fc46748399af0875eea4111e8" gracePeriod=30 Oct 30 01:22:00.405128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.405102 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:22:00.451287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451251 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/345e5ad3-c558-478f-a4a9-641dec5af039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345e5ad3-c558-478f-a4a9-641dec5af039-istio-podinfo\") pod \"345e5ad3-c558-478f-a4a9-641dec5af039\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " Oct 30 01:22:00.451427 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451367 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-socket\") pod \"345e5ad3-c558-478f-a4a9-641dec5af039\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " Oct 30 01:22:00.451427 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451398 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-istio-token\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-istio-token\") pod \"345e5ad3-c558-478f-a4a9-641dec5af039\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " Oct 30 01:22:00.451521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451429 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-kube-api-access-qxrtk\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-kube-api-access-qxrtk\") pod \"345e5ad3-c558-478f-a4a9-641dec5af039\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " Oct 30 01:22:00.451521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451474 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/345e5ad3-c558-478f-a4a9-641dec5af039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345e5ad3-c558-478f-a4a9-641dec5af039-istiod-ca-cert\") pod \"345e5ad3-c558-478f-a4a9-641dec5af039\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " Oct 30 01:22:00.451644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451554 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-data\") pod \"345e5ad3-c558-478f-a4a9-641dec5af039\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " Oct 30 01:22:00.451776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451747 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-socket" pod "345e5ad3-c558-478f-a4a9-641dec5af039" (UID: "345e5ad3-c558-478f-a4a9-641dec5af039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:00.451862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451766 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-credential-socket\") pod \"345e5ad3-c558-478f-a4a9-641dec5af039\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " Oct 30 01:22:00.451921 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451852 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-data" pod "345e5ad3-c558-478f-a4a9-641dec5af039" (UID: "345e5ad3-c558-478f-a4a9-641dec5af039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:00.451993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.451970 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-envoy\") pod \"345e5ad3-c558-478f-a4a9-641dec5af039\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " Oct 30 01:22:00.452135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.452002 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-credential-socket" pod "345e5ad3-c558-478f-a4a9-641dec5af039" (UID: "345e5ad3-c558-478f-a4a9-641dec5af039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:00.452135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.452008 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-certs\") pod \"345e5ad3-c558-478f-a4a9-641dec5af039\" (UID: \"345e5ad3-c558-478f-a4a9-641dec5af039\") " Oct 30 01:22:00.452135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.452023 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345e5ad3-c558-478f-a4a9-641dec5af039-istiod-ca-cert" pod "345e5ad3-c558-478f-a4a9-641dec5af039" (UID: "345e5ad3-c558-478f-a4a9-641dec5af039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:00.452319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.452219 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-certs" pod "345e5ad3-c558-478f-a4a9-641dec5af039" (UID: "345e5ad3-c558-478f-a4a9-641dec5af039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:00.452858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.452832 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:00.452947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.452863 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345e5ad3-c558-478f-a4a9-641dec5af039-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:00.452947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.452878 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:00.452947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.452891 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:00.452947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.452906 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:00.453843 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.453816 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/345e5ad3-c558-478f-a4a9-641dec5af039-istio-podinfo" pod "345e5ad3-c558-478f-a4a9-641dec5af039" (UID: "345e5ad3-c558-478f-a4a9-641dec5af039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:00.453934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.453920 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-envoy" pod "345e5ad3-c558-478f-a4a9-641dec5af039" (UID: "345e5ad3-c558-478f-a4a9-641dec5af039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:00.454114 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.454094 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-kube-api-access-qxrtk" pod "345e5ad3-c558-478f-a4a9-641dec5af039" (UID: "345e5ad3-c558-478f-a4a9-641dec5af039"). InnerVolumeSpecName "kube-api-access-qxrtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:00.454230 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.454208 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-istio-token" pod "345e5ad3-c558-478f-a4a9-641dec5af039" (UID: "345e5ad3-c558-478f-a4a9-641dec5af039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:00.553912 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.553882 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345e5ad3-c558-478f-a4a9-641dec5af039-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:00.553912 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.553909 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:00.553912 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.553919 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxrtk\" (UniqueName: \"kubernetes.io/projected/345e5ad3-c558-478f-a4a9-641dec5af039-kube-api-access-qxrtk\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:00.554147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:00.553930 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345e5ad3-c558-478f-a4a9-641dec5af039-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:01.052334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.052294 4927 generic.go:356] "Generic (PLEG): container finished" podID="345e5ad3-c558-478f-a4a9-641dec5af039" containerID="fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8" exitCode=0 Oct 30 01:22:01.052518 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.052348 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" Oct 30 01:22:01.052518 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.052368 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" event={"ID":"345e5ad3-c558-478f-a4a9-641dec5af039","Type":"ContainerDied","Data":"fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8"} Oct 30 01:22:01.052518 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.052438 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm" event={"ID":"345e5ad3-c558-478f-a4a9-641dec5af039","Type":"ContainerDied","Data":"6f990dd91695d4095bafd8658f78dd1f77b39577237c4788b4bc8471ed5ccb79"} Oct 30 01:22:01.052518 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.052470 4927 scope.go:122] "RemoveContainer" containerID="fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8" Oct 30 01:22:01.064639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.064614 4927 scope.go:122] "RemoveContainer" containerID="a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e" Oct 30 01:22:01.075319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.075295 4927 scope.go:122] "RemoveContainer" containerID="fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8" Oct 30 01:22:01.075611 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:01.075559 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8\": container with ID starting with fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8 not found: ID does not exist" containerID="fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8" Oct 30 01:22:01.075692 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.075621 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8"} err="failed to get container status \"fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8\": rpc error: code = NotFound desc = could not find container \"fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8\": container with ID starting with fccbfcd8414cf25751a36e143c4a753cc6bad32bfec3478e219e3a5b730cb5a8 not found: ID does not exist" Oct 30 01:22:01.075692 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.075641 4927 scope.go:122] "RemoveContainer" containerID="a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e" Oct 30 01:22:01.075932 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:01.075912 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e\": container with ID starting with a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e not found: ID does not exist" containerID="a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e" Oct 30 01:22:01.075988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.075941 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e"} err="failed to get container status \"a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e\": rpc error: code = NotFound desc = could not find container \"a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e\": container with ID starting with a1a9ac66a2bc3aaf371327c864a5fba9c5a9df259fbf10b77d89fa355f86909e not found: ID does not exist" Oct 30 01:22:01.116918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.116890 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm"] Oct 30 01:22:01.127965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.127939 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d1ae36-openshift-default-56d5b87b55-n9plm"] Oct 30 01:22:01.443943 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.443907 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:22:01.444333 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.443993 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:22:01.574787 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.574754 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" path="/var/lib/kubelet/pods/345e5ad3-c558-478f-a4a9-641dec5af039/volumes" Oct 30 01:22:01.931303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.931265 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:22:01.931523 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:01.931350 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:22:02.950016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.949824 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48"] Oct 30 01:22:02.951471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.951104 4927 cpu_manager.go:414] "containerMap: removing container" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" Oct 30 01:22:02.951471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.951130 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" Oct 30 01:22:02.951471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.951175 4927 cpu_manager.go:414] "containerMap: removing container" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" Oct 30 01:22:02.951471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.951183 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" Oct 30 01:22:02.951471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.951367 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" Oct 30 01:22:02.951471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.951390 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="345e5ad3-c558-478f-a4a9-641dec5af039" containerName="istio-proxy" Oct 30 01:22:02.959566 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.959538 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:02.969247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.969219 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48"] Oct 30 01:22:02.988839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.988631 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6cba49-dockercfg-ptvqv\"" Oct 30 01:22:02.988839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:02.988743 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6cba49\"" Oct 30 01:22:03.098063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.098030 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c"] Oct 30 01:22:03.102615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.102591 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:03.110458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.110426 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxst2\" (UniqueName: \"kubernetes.io/projected/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-kube-api-access-dxst2\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48\" (UID: \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:03.110631 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.110494 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48\" (UID: \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:03.123245 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.123215 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c"] Oct 30 01:22:03.211640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.211527 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48\" (UID: \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:03.211828 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.211649 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47739eb1-7fec-48dc-91c7-405c4ecc53c8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c\" (UID: \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:03.211828 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.211703 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xdvb\" (UniqueName: \"kubernetes.io/projected/47739eb1-7fec-48dc-91c7-405c4ecc53c8-kube-api-access-2xdvb\") pod \"payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c\" (UID: \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:03.211963 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.211875 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxst2\" (UniqueName: \"kubernetes.io/projected/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-kube-api-access-dxst2\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48\" (UID: \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:03.212363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.212337 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48\" (UID: \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:03.234760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.234732 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxst2\" (UniqueName: \"kubernetes.io/projected/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-kube-api-access-dxst2\") pod \"payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48\" (UID: \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:03.290368 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.290332 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:03.313057 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.313029 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47739eb1-7fec-48dc-91c7-405c4ecc53c8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c\" (UID: \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:03.313185 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.313095 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdvb\" (UniqueName: \"kubernetes.io/projected/47739eb1-7fec-48dc-91c7-405c4ecc53c8-kube-api-access-2xdvb\") pod \"payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c\" (UID: \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:03.313994 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.313967 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47739eb1-7fec-48dc-91c7-405c4ecc53c8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c\" (UID: \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:03.335997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.335929 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdvb\" (UniqueName: \"kubernetes.io/projected/47739eb1-7fec-48dc-91c7-405c4ecc53c8-kube-api-access-2xdvb\") pod \"payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c\" (UID: \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:03.419853 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.419811 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:03.604886 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.604857 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48"] Oct 30 01:22:03.606434 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:22:03.606407 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e24a86_fcf6_4532_b9fd_7dc8b33d374e.slice/crio-1a02ad7b5dc50332318603d47d599deb449ec15ea899a9c792378643545b3a01 WatchSource:0}: Error finding container 1a02ad7b5dc50332318603d47d599deb449ec15ea899a9c792378643545b3a01: Status 404 returned error can't find the container with id 1a02ad7b5dc50332318603d47d599deb449ec15ea899a9c792378643545b3a01 Oct 30 01:22:03.755599 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:22:03.755307 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47739eb1_7fec_48dc_91c7_405c4ecc53c8.slice/crio-dc2fa3ee7d592e047977ddd5999c6cbf1711666b588f8f83e637a18db7433e24 WatchSource:0}: Error finding container dc2fa3ee7d592e047977ddd5999c6cbf1711666b588f8f83e637a18db7433e24: Status 404 returned error can't find the container with id dc2fa3ee7d592e047977ddd5999c6cbf1711666b588f8f83e637a18db7433e24 Oct 30 01:22:03.756499 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:03.756222 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c"] Oct 30 01:22:04.082334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.082299 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" event={"ID":"60e24a86-fcf6-4532-b9fd-7dc8b33d374e","Type":"ContainerStarted","Data":"e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab"} Oct 30 01:22:04.082840 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.082351 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" event={"ID":"60e24a86-fcf6-4532-b9fd-7dc8b33d374e","Type":"ContainerStarted","Data":"1a02ad7b5dc50332318603d47d599deb449ec15ea899a9c792378643545b3a01"} Oct 30 01:22:04.083345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.083322 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:04.086376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.086272 4927 generic.go:356] "Generic (PLEG): container finished" podID="83ebc161-dea3-44cb-845b-8531f0399603" containerID="179ee5373677eff984a1961d70f2ae2d9355ea8fc46748399af0875eea4111e8" exitCode=0 Oct 30 01:22:04.086376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.086322 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" event={"ID":"83ebc161-dea3-44cb-845b-8531f0399603","Type":"ContainerDied","Data":"179ee5373677eff984a1961d70f2ae2d9355ea8fc46748399af0875eea4111e8"} Oct 30 01:22:04.089700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.089669 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" event={"ID":"47739eb1-7fec-48dc-91c7-405c4ecc53c8","Type":"ContainerStarted","Data":"cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139"} Oct 30 01:22:04.089818 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.089729 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" event={"ID":"47739eb1-7fec-48dc-91c7-405c4ecc53c8","Type":"ContainerStarted","Data":"dc2fa3ee7d592e047977ddd5999c6cbf1711666b588f8f83e637a18db7433e24"} Oct 30 01:22:04.089881 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.089839 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:04.107352 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.107327 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:22:04.123016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.122973 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" podStartSLOduration=2.122948915 podStartE2EDuration="2.122948915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:04.121249345 +0000 UTC m=+2862.693175571" watchObservedRunningTime="2026-10-30 01:22:04.122948915 +0000 UTC m=+2862.694875137" Oct 30 01:22:04.164825 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.164770 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" podStartSLOduration=1.164756069 podStartE2EDuration="1.164756069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:04.160008692 +0000 UTC m=+2862.731934914" watchObservedRunningTime="2026-10-30 01:22:04.164756069 +0000 UTC m=+2862.736682290" Oct 30 01:22:04.230180 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230146 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ebc161-dea3-44cb-845b-8531f0399603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebc161-dea3-44cb-845b-8531f0399603-istiod-ca-cert\") pod \"83ebc161-dea3-44cb-845b-8531f0399603\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " Oct 30 01:22:04.230389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230207 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-envoy\") pod \"83ebc161-dea3-44cb-845b-8531f0399603\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " Oct 30 01:22:04.230389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230372 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-kube-api-access-mj6tj\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-kube-api-access-mj6tj\") pod \"83ebc161-dea3-44cb-845b-8531f0399603\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " Oct 30 01:22:04.230505 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230408 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83ebc161-dea3-44cb-845b-8531f0399603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebc161-dea3-44cb-845b-8531f0399603-istio-podinfo\") pod \"83ebc161-dea3-44cb-845b-8531f0399603\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " Oct 30 01:22:04.230505 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230461 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-istio-token\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-istio-token\") pod \"83ebc161-dea3-44cb-845b-8531f0399603\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " Oct 30 01:22:04.230636 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230508 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-data\") pod \"83ebc161-dea3-44cb-845b-8531f0399603\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " Oct 30 01:22:04.230636 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230627 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-certs\") pod \"83ebc161-dea3-44cb-845b-8531f0399603\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " Oct 30 01:22:04.230746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230652 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ebc161-dea3-44cb-845b-8531f0399603-istiod-ca-cert" pod "83ebc161-dea3-44cb-845b-8531f0399603" (UID: "83ebc161-dea3-44cb-845b-8531f0399603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:04.230746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230663 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-socket\") pod \"83ebc161-dea3-44cb-845b-8531f0399603\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " Oct 30 01:22:04.230746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.230697 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-credential-socket\") pod \"83ebc161-dea3-44cb-845b-8531f0399603\" (UID: \"83ebc161-dea3-44cb-845b-8531f0399603\") " Oct 30 01:22:04.232136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.231244 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-certs" pod "83ebc161-dea3-44cb-845b-8531f0399603" (UID: "83ebc161-dea3-44cb-845b-8531f0399603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.232136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.231260 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-socket" pod "83ebc161-dea3-44cb-845b-8531f0399603" (UID: "83ebc161-dea3-44cb-845b-8531f0399603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.232136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.231487 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-credential-socket" pod "83ebc161-dea3-44cb-845b-8531f0399603" (UID: "83ebc161-dea3-44cb-845b-8531f0399603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.232136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.231509 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-data" pod "83ebc161-dea3-44cb-845b-8531f0399603" (UID: "83ebc161-dea3-44cb-845b-8531f0399603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.232136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.231755 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.232136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.231776 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.232136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.231788 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.232136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.231804 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebc161-dea3-44cb-845b-8531f0399603-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.232136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.231819 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.233794 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.233767 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-envoy" pod "83ebc161-dea3-44cb-845b-8531f0399603" (UID: "83ebc161-dea3-44cb-845b-8531f0399603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:04.234004 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.233980 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-istio-token" pod "83ebc161-dea3-44cb-845b-8531f0399603" (UID: "83ebc161-dea3-44cb-845b-8531f0399603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:04.235498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.235470 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83ebc161-dea3-44cb-845b-8531f0399603-istio-podinfo" pod "83ebc161-dea3-44cb-845b-8531f0399603" (UID: "83ebc161-dea3-44cb-845b-8531f0399603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:04.235995 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.235975 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-kube-api-access-mj6tj" pod "83ebc161-dea3-44cb-845b-8531f0399603" (UID: "83ebc161-dea3-44cb-845b-8531f0399603"). InnerVolumeSpecName "kube-api-access-mj6tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:04.301161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.301069 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6"] Oct 30 01:22:04.335853 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.335723 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebc161-dea3-44cb-845b-8531f0399603-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.335853 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.335762 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj6tj\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-kube-api-access-mj6tj\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.335853 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.335779 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebc161-dea3-44cb-845b-8531f0399603-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.335853 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:04.335796 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebc161-dea3-44cb-845b-8531f0399603-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.450704 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:04.450664 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:22:05.100246 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.100215 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" Oct 30 01:22:05.100246 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.100239 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv" event={"ID":"83ebc161-dea3-44cb-845b-8531f0399603","Type":"ContainerDied","Data":"caedd7c80388e71a71c4d8dc25708f3cc0b2c6beb4a517321a9d06ae9762745c"} Oct 30 01:22:05.100885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.100307 4927 scope.go:122] "RemoveContainer" containerID="179ee5373677eff984a1961d70f2ae2d9355ea8fc46748399af0875eea4111e8" Oct 30 01:22:05.169279 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.169241 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv"] Oct 30 01:22:05.173663 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.173634 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-2dghv"] Oct 30 01:22:05.563666 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.563619 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m"] Oct 30 01:22:05.565364 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.564918 4927 cpu_manager.go:414] "containerMap: removing container" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" Oct 30 01:22:05.565364 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.564948 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" Oct 30 01:22:05.565364 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.565226 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ebc161-dea3-44cb-845b-8531f0399603" containerName="istio-proxy" Oct 30 01:22:05.580206 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.580169 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:05.597092 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.597051 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fc3cb2\"" Oct 30 01:22:05.597310 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.597254 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fc3cb2-dockercfg-2w4hs\"" Oct 30 01:22:05.599824 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.599798 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ebc161-dea3-44cb-845b-8531f0399603" path="/var/lib/kubelet/pods/83ebc161-dea3-44cb-845b-8531f0399603/volumes" Oct 30 01:22:05.600273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.600252 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m"] Oct 30 01:22:05.633805 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.633776 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7"] Oct 30 01:22:05.640990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.640965 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:05.647748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.647130 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7"] Oct 30 01:22:05.680140 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.680104 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3987a5c-3a1d-4602-af47-7fb88849fa9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m\" (UID: \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:05.680329 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.680211 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwcl\" (UniqueName: \"kubernetes.io/projected/b3987a5c-3a1d-4602-af47-7fb88849fa9e-kube-api-access-6dwcl\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m\" (UID: \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:05.782034 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.781998 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n699m\" (UniqueName: \"kubernetes.io/projected/23d22d03-5652-4aa6-a40c-6de31a390844-kube-api-access-n699m\") pod \"payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7\" (UID: \"23d22d03-5652-4aa6-a40c-6de31a390844\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:05.782256 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.782083 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3987a5c-3a1d-4602-af47-7fb88849fa9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m\" (UID: \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:05.782256 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.782164 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d22d03-5652-4aa6-a40c-6de31a390844-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7\" (UID: \"23d22d03-5652-4aa6-a40c-6de31a390844\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:05.782395 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.782321 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwcl\" (UniqueName: \"kubernetes.io/projected/b3987a5c-3a1d-4602-af47-7fb88849fa9e-kube-api-access-6dwcl\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m\" (UID: \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:05.785446 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.784701 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3987a5c-3a1d-4602-af47-7fb88849fa9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m\" (UID: \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:05.805880 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.805855 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwcl\" (UniqueName: \"kubernetes.io/projected/b3987a5c-3a1d-4602-af47-7fb88849fa9e-kube-api-access-6dwcl\") pod \"payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m\" (UID: \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:05.885146 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.885113 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n699m\" (UniqueName: \"kubernetes.io/projected/23d22d03-5652-4aa6-a40c-6de31a390844-kube-api-access-n699m\") pod \"payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7\" (UID: \"23d22d03-5652-4aa6-a40c-6de31a390844\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:05.885446 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.885425 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d22d03-5652-4aa6-a40c-6de31a390844-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7\" (UID: \"23d22d03-5652-4aa6-a40c-6de31a390844\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:05.886275 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.886255 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d22d03-5652-4aa6-a40c-6de31a390844-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7\" (UID: \"23d22d03-5652-4aa6-a40c-6de31a390844\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:05.907956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.907932 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n699m\" (UniqueName: \"kubernetes.io/projected/23d22d03-5652-4aa6-a40c-6de31a390844-kube-api-access-n699m\") pod \"payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7\" (UID: \"23d22d03-5652-4aa6-a40c-6de31a390844\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:05.934777 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.934745 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:05.962716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:05.962082 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:06.290231 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.290196 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m"] Oct 30 01:22:06.293152 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:22:06.293121 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3987a5c_3a1d_4602_af47_7fb88849fa9e.slice/crio-08bd5d8b52146bc27439058d6944ae2a0406e0f2aeebe9aba34e9197cd9fecfc WatchSource:0}: Error finding container 08bd5d8b52146bc27439058d6944ae2a0406e0f2aeebe9aba34e9197cd9fecfc: Status 404 returned error can't find the container with id 08bd5d8b52146bc27439058d6944ae2a0406e0f2aeebe9aba34e9197cd9fecfc Oct 30 01:22:06.310963 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.307142 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7"] Oct 30 01:22:06.325456 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:22:06.325406 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d22d03_5652_4aa6_a40c_6de31a390844.slice/crio-fb326a879cc4ee8c535680a849e0a0324422432b6928907711fadd2bfd0bda41 WatchSource:0}: Error finding container fb326a879cc4ee8c535680a849e0a0324422432b6928907711fadd2bfd0bda41: Status 404 returned error can't find the container with id fb326a879cc4ee8c535680a849e0a0324422432b6928907711fadd2bfd0bda41 Oct 30 01:22:06.429696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.429155 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2"] Oct 30 01:22:06.429696 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.429429 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" containerID="cri-o://44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c" gracePeriod=30 Oct 30 01:22:06.849928 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.849889 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7"] Oct 30 01:22:06.856235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.856203 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:06.875051 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.873533 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3e9921-openshift-default-dockercfg-96nf7\"" Oct 30 01:22:06.880916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.880886 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7"] Oct 30 01:22:06.908999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.908218 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-socket\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:06.908999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.908287 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-certs\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:06.908999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.908369 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4836cefc-d5ca-469d-899b-d9298b44db87-istiod-ca-cert\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:06.908999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.908486 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4836cefc-d5ca-469d-899b-d9298b44db87-istio-podinfo\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:06.908999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.908512 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qptcd\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-kube-api-access-qptcd\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:06.908999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.908566 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-envoy\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:06.908999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.908625 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-data\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:06.908999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.908662 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-credential-socket\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:06.908999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:06.908696 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-istio-token\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.010427 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.010149 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4836cefc-d5ca-469d-899b-d9298b44db87-istio-podinfo\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.010890 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.010841 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qptcd\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-kube-api-access-qptcd\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.011315 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-envoy\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.011367 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-data\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.011433 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-credential-socket\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.011472 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-istio-token\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.011681 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-socket\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.011714 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-certs\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.011806 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4836cefc-d5ca-469d-899b-d9298b44db87-istiod-ca-cert\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.012744 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4836cefc-d5ca-469d-899b-d9298b44db87-istiod-ca-cert\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.013657 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4836cefc-d5ca-469d-899b-d9298b44db87-istio-podinfo\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.013844 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-credential-socket\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.013933 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-socket\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.025757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.014089 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-certs\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.028385 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.028154 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-data\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.059591 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.039006 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-envoy\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.069738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.069669 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qptcd\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-kube-api-access-qptcd\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.075161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.075097 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-istio-token\") pod \"e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.208590 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.202536 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:07.229388 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.229068 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" event={"ID":"b3987a5c-3a1d-4602-af47-7fb88849fa9e","Type":"ContainerStarted","Data":"8940af674b3eafb1f2a521107c43951199fcec1ab6a1a2c6b676d533b1e1a2bb"} Oct 30 01:22:07.229388 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.229132 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" event={"ID":"b3987a5c-3a1d-4602-af47-7fb88849fa9e","Type":"ContainerStarted","Data":"08bd5d8b52146bc27439058d6944ae2a0406e0f2aeebe9aba34e9197cd9fecfc"} Oct 30 01:22:07.231230 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.231199 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" event={"ID":"23d22d03-5652-4aa6-a40c-6de31a390844","Type":"ContainerStarted","Data":"31e1b784f841b199715c38db71fe0f3284a7a658e04aac18bed842f0c65c1492"} Oct 30 01:22:07.231352 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.231259 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" event={"ID":"23d22d03-5652-4aa6-a40c-6de31a390844","Type":"ContainerStarted","Data":"fb326a879cc4ee8c535680a849e0a0324422432b6928907711fadd2bfd0bda41"} Oct 30 01:22:07.232000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.231974 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:07.278625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.274915 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" podStartSLOduration=2.2748928409999998 podStartE2EDuration="2.274892841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:07.269556925 +0000 UTC m=+2865.841483148" watchObservedRunningTime="2026-10-30 01:22:07.274892841 +0000 UTC m=+2865.846819065" Oct 30 01:22:07.318719 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.315363 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" podStartSLOduration=2.3153418119999998 podStartE2EDuration="2.315341812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:07.30891566 +0000 UTC m=+2865.880841884" watchObservedRunningTime="2026-10-30 01:22:07.315341812 +0000 UTC m=+2865.887268036" Oct 30 01:22:07.600104 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:22:07.600070 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4836cefc_d5ca_469d_899b_d9298b44db87.slice/crio-8e82ae47ca00b12528df16e90f1c12a190b4a73d02d940ebe8bd72971b04fb2c WatchSource:0}: Error finding container 8e82ae47ca00b12528df16e90f1c12a190b4a73d02d940ebe8bd72971b04fb2c: Status 404 returned error can't find the container with id 8e82ae47ca00b12528df16e90f1c12a190b4a73d02d940ebe8bd72971b04fb2c Oct 30 01:22:07.601494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.601464 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7"] Oct 30 01:22:07.604390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.604349 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:22:07.604507 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.604443 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:22:07.604507 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:07.604494 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:22:08.241777 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:08.241741 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" event={"ID":"4836cefc-d5ca-469d-899b-d9298b44db87","Type":"ContainerStarted","Data":"3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426"} Oct 30 01:22:08.241985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:08.241806 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" event={"ID":"4836cefc-d5ca-469d-899b-d9298b44db87","Type":"ContainerStarted","Data":"8e82ae47ca00b12528df16e90f1c12a190b4a73d02d940ebe8bd72971b04fb2c"} Oct 30 01:22:08.242180 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:08.242158 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:08.292077 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:08.292013 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" podStartSLOduration=2.291992876 podStartE2EDuration="2.291992876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:08.285601306 +0000 UTC m=+2866.857527532" watchObservedRunningTime="2026-10-30 01:22:08.291992876 +0000 UTC m=+2866.863919101" Oct 30 01:22:09.204059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:09.204022 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:09.211197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:09.211169 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:09.250212 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:09.250179 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:09.251241 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:09.251215 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:22:10.443050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:10.443011 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd"] Oct 30 01:22:10.482975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:10.482933 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86"] Oct 30 01:22:11.223734 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.223692 4927 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:22:11.224020 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.223992 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:22:11.656750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.656722 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:22:11.716641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.716602 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-kube-api-access-xjtql\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-kube-api-access-xjtql\") pod \"a3afe368-c954-42cd-8d8b-6753ebab041d\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " Oct 30 01:22:11.716854 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.716669 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-socket\") pod \"a3afe368-c954-42cd-8d8b-6753ebab041d\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " Oct 30 01:22:11.716854 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.716725 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-certs\") pod \"a3afe368-c954-42cd-8d8b-6753ebab041d\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " Oct 30 01:22:11.716854 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.716812 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-token\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-token\") pod \"a3afe368-c954-42cd-8d8b-6753ebab041d\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " Oct 30 01:22:11.717061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.716881 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3afe368-c954-42cd-8d8b-6753ebab041d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afe368-c954-42cd-8d8b-6753ebab041d-istiod-ca-cert\") pod \"a3afe368-c954-42cd-8d8b-6753ebab041d\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " Oct 30 01:22:11.717061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.716955 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-data\") pod \"a3afe368-c954-42cd-8d8b-6753ebab041d\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " Oct 30 01:22:11.717061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.717056 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-credential-socket\") pod \"a3afe368-c954-42cd-8d8b-6753ebab041d\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " Oct 30 01:22:11.717224 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.717079 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-podinfo\") pod \"a3afe368-c954-42cd-8d8b-6753ebab041d\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " Oct 30 01:22:11.717224 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.717186 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-envoy\") pod \"a3afe368-c954-42cd-8d8b-6753ebab041d\" (UID: \"a3afe368-c954-42cd-8d8b-6753ebab041d\") " Oct 30 01:22:11.718563 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.718328 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-socket" pod "a3afe368-c954-42cd-8d8b-6753ebab041d" (UID: "a3afe368-c954-42cd-8d8b-6753ebab041d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:11.718563 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.718369 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-certs" pod "a3afe368-c954-42cd-8d8b-6753ebab041d" (UID: "a3afe368-c954-42cd-8d8b-6753ebab041d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:11.718563 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.718402 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-credential-socket" pod "a3afe368-c954-42cd-8d8b-6753ebab041d" (UID: "a3afe368-c954-42cd-8d8b-6753ebab041d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:11.718563 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.718424 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-data" pod "a3afe368-c954-42cd-8d8b-6753ebab041d" (UID: "a3afe368-c954-42cd-8d8b-6753ebab041d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:11.718849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.718702 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3afe368-c954-42cd-8d8b-6753ebab041d-istiod-ca-cert" pod "a3afe368-c954-42cd-8d8b-6753ebab041d" (UID: "a3afe368-c954-42cd-8d8b-6753ebab041d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:11.720621 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.720591 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-podinfo" pod "a3afe368-c954-42cd-8d8b-6753ebab041d" (UID: "a3afe368-c954-42cd-8d8b-6753ebab041d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:11.720839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.720809 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-envoy" pod "a3afe368-c954-42cd-8d8b-6753ebab041d" (UID: "a3afe368-c954-42cd-8d8b-6753ebab041d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:11.720956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.720861 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-kube-api-access-xjtql" pod "a3afe368-c954-42cd-8d8b-6753ebab041d" (UID: "a3afe368-c954-42cd-8d8b-6753ebab041d"). InnerVolumeSpecName "kube-api-access-xjtql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:11.720956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.720885 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-token" pod "a3afe368-c954-42cd-8d8b-6753ebab041d" (UID: "a3afe368-c954-42cd-8d8b-6753ebab041d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:11.818613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.818515 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:11.818613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.818541 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afe368-c954-42cd-8d8b-6753ebab041d-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:11.818613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.818550 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:11.818613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.818560 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:11.818613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.818582 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:11.818613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.818596 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:11.818613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.818604 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjtql\" (UniqueName: \"kubernetes.io/projected/a3afe368-c954-42cd-8d8b-6753ebab041d-kube-api-access-xjtql\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:11.818613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.818614 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:11.819014 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.818625 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afe368-c954-42cd-8d8b-6753ebab041d-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:11.877444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.877401 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s"] Oct 30 01:22:11.884312 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:11.884283 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v"] Oct 30 01:22:12.282943 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.282718 4927 generic.go:356] "Generic (PLEG): container finished" podID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerID="44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c" exitCode=0 Oct 30 01:22:12.282943 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.282821 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" event={"ID":"a3afe368-c954-42cd-8d8b-6753ebab041d","Type":"ContainerDied","Data":"44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c"} Oct 30 01:22:12.282943 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.282882 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" event={"ID":"a3afe368-c954-42cd-8d8b-6753ebab041d","Type":"ContainerDied","Data":"93a184eab0ace4998ce4de4e6eb07ed1c45a7582ebf05e50e88da5842e58e2f2"} Oct 30 01:22:12.282943 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.282914 4927 scope.go:122] "RemoveContainer" containerID="44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c" Oct 30 01:22:12.283337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.283100 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2" Oct 30 01:22:12.301566 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.301537 4927 scope.go:122] "RemoveContainer" containerID="44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c" Oct 30 01:22:12.301896 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:12.301870 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c\": container with ID starting with 44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c not found: ID does not exist" containerID="44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c" Oct 30 01:22:12.301974 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.301916 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c"} err="failed to get container status \"44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c\": rpc error: code = NotFound desc = could not find container \"44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c\": container with ID starting with 44c3dde79b02dbe4fdd92f66e649d0963c1910660719b711f27f51dc4a7d285c not found: ID does not exist" Oct 30 01:22:12.347212 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.347181 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2"] Oct 30 01:22:12.351792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.351763 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-986c2"] Oct 30 01:22:12.507918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.507884 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x"] Oct 30 01:22:12.508957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.508924 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" Oct 30 01:22:12.508957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.508954 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" Oct 30 01:22:12.509128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.509079 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" containerName="istio-proxy" Oct 30 01:22:12.514446 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.514420 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:12.527888 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.527860 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599b6-0434-4304-be95-553cd7dd0ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x\" (UID: \"ac0599b6-0434-4304-be95-553cd7dd0ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:12.528014 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.528000 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-778b8\" (UniqueName: \"kubernetes.io/projected/ac0599b6-0434-4304-be95-553cd7dd0ba3-kube-api-access-778b8\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x\" (UID: \"ac0599b6-0434-4304-be95-553cd7dd0ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:12.529911 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.529884 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3e9921\"" Oct 30 01:22:12.530207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.530171 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3e9921-dockercfg-f782b\"" Oct 30 01:22:12.535226 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.535163 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x"] Oct 30 01:22:12.705628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.655879 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599b6-0434-4304-be95-553cd7dd0ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x\" (UID: \"ac0599b6-0434-4304-be95-553cd7dd0ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:12.705628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.656059 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-778b8\" (UniqueName: \"kubernetes.io/projected/ac0599b6-0434-4304-be95-553cd7dd0ba3-kube-api-access-778b8\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x\" (UID: \"ac0599b6-0434-4304-be95-553cd7dd0ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:12.706495 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.706428 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599b6-0434-4304-be95-553cd7dd0ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x\" (UID: \"ac0599b6-0434-4304-be95-553cd7dd0ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:12.727015 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.726985 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-778b8\" (UniqueName: \"kubernetes.io/projected/ac0599b6-0434-4304-be95-553cd7dd0ba3-kube-api-access-778b8\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x\" (UID: \"ac0599b6-0434-4304-be95-553cd7dd0ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:12.836862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:12.836776 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:13.152899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:13.152855 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x"] Oct 30 01:22:13.153123 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:22:13.153100 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0599b6_0434_4304_be95_553cd7dd0ba3.slice/crio-768106c9116fb8e8622aebdadd1e0017d5c9ae1b0956f682e631e60233a4a10a WatchSource:0}: Error finding container 768106c9116fb8e8622aebdadd1e0017d5c9ae1b0956f682e631e60233a4a10a: Status 404 returned error can't find the container with id 768106c9116fb8e8622aebdadd1e0017d5c9ae1b0956f682e631e60233a4a10a Oct 30 01:22:13.296179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:13.296116 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" event={"ID":"ac0599b6-0434-4304-be95-553cd7dd0ba3","Type":"ContainerStarted","Data":"49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f"} Oct 30 01:22:13.296179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:13.296171 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" event={"ID":"ac0599b6-0434-4304-be95-553cd7dd0ba3","Type":"ContainerStarted","Data":"768106c9116fb8e8622aebdadd1e0017d5c9ae1b0956f682e631e60233a4a10a"} Oct 30 01:22:13.296427 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:13.296203 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:13.349900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:13.349834 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" podStartSLOduration=1.349817424 podStartE2EDuration="1.349817424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:13.34578759 +0000 UTC m=+2871.917713813" watchObservedRunningTime="2026-10-30 01:22:13.349817424 +0000 UTC m=+2871.921743646" Oct 30 01:22:13.573340 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:13.573262 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3afe368-c954-42cd-8d8b-6753ebab041d" path="/var/lib/kubelet/pods/a3afe368-c954-42cd-8d8b-6753ebab041d/volumes" Oct 30 01:22:15.103706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.103677 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:15.443382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.443330 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" podUID="789da654-992f-48cc-80d2-5b069ed77feb" containerName="payload-processing" containerID="cri-o://eae39647c935f34c51c0be9ece233978850257cec10d101966142e1303efda9a" gracePeriod=25 Oct 30 01:22:15.483289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.483237 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" podUID="bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" containerName="payload-pre-processing" containerID="cri-o://68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66" gracePeriod=25 Oct 30 01:22:15.790301 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.790221 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c"] Oct 30 01:22:15.796184 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.796154 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:15.826700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.826651 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5f99e7-openshift-default-dockercfg-lqqpl\"" Oct 30 01:22:15.833314 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.833277 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c"] Oct 30 01:22:15.910740 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.910705 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-podinfo\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:15.910921 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.910747 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-data\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:15.910963 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.910922 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-credential-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:15.910963 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.910951 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-envoy\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:15.911044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.910971 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istiod-ca-cert\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:15.911134 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.911106 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-certs\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:15.911191 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.911157 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:15.911272 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.911242 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-token\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:15.911396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:15.911379 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xc2m\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-kube-api-access-4xc2m\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.012992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.012954 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-certs\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.012992 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.012995 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013023 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-token\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013097 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xc2m\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-kube-api-access-4xc2m\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013171 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-podinfo\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013292 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013186 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-data\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013337 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-credential-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013372 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-envoy\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013400 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istiod-ca-cert\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013481 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013725 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013625 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-data\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013725 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013647 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-certs\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.013725 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.013702 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-credential-socket\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.014138 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.014120 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istiod-ca-cert\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.015660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.015633 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-envoy\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.015935 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.015914 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-podinfo\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.035553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.035522 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-token\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.035706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.035629 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xc2m\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-kube-api-access-4xc2m\") pod \"e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.112115 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.112031 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:16.120320 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.120290 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:16.449426 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.449384 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c"] Oct 30 01:22:16.449682 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:22:16.449649 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7e7ec9_3153_438f_820b_c3e8e35bd040.slice/crio-90cd243f270294c5e35c3f5229cead43dea8e34901d238111a22b1d611f0a755 WatchSource:0}: Error finding container 90cd243f270294c5e35c3f5229cead43dea8e34901d238111a22b1d611f0a755: Status 404 returned error can't find the container with id 90cd243f270294c5e35c3f5229cead43dea8e34901d238111a22b1d611f0a755 Oct 30 01:22:16.452541 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.452505 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:22:16.452624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.452593 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:22:16.452663 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:16.452629 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:22:17.335107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:17.335066 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" event={"ID":"2a7e7ec9-3153-438f-820b-c3e8e35bd040","Type":"ContainerStarted","Data":"e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0"} Oct 30 01:22:17.335540 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:17.335134 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" event={"ID":"2a7e7ec9-3153-438f-820b-c3e8e35bd040","Type":"ContainerStarted","Data":"90cd243f270294c5e35c3f5229cead43dea8e34901d238111a22b1d611f0a755"} Oct 30 01:22:17.429905 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:17.429848 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" podStartSLOduration=2.429833241 podStartE2EDuration="2.429833241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:17.415175134 +0000 UTC m=+2875.987101381" watchObservedRunningTime="2026-10-30 01:22:17.429833241 +0000 UTC m=+2876.001759462" Oct 30 01:22:18.117608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:18.115988 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:18.117608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:18.116211 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 01:22:18.117608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:18.116273 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 01:22:19.112747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.112708 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 01:22:19.113169 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.112778 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 01:22:19.252423 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.252387 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:22:19.252690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.252474 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:22:19.358102 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.358063 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" containerID="cri-o://1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc" gracePeriod=45 Oct 30 01:22:19.722305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.722274 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 01:22:19.800642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.778124 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-kube-api-access-ns5bm\" (UniqueName: \"kubernetes.io/projected/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-kube-api-access-ns5bm\") pod \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " Oct 30 01:22:19.801062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.801031 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-dshm\") pod \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " Oct 30 01:22:19.801245 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.801229 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-home\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-home\") pod \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " Oct 30 01:22:19.801382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.801366 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-model-cache\") pod \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " Oct 30 01:22:19.801610 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.801592 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tmp-dir\") pod \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " Oct 30 01:22:19.801871 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.801855 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tls-certs\") pod \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\" (UID: \"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7\") " Oct 30 01:22:19.806741 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.806360 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-home" pod "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" (UID: "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.807298 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.807258 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tmp-dir" pod "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" (UID: "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.808022 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.808001 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.808305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.808291 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.810410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.810377 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-dshm" pod "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" (UID: "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.810528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.810480 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tls-certs" pod "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" (UID: "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:19.810799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.810766 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-model-cache" pod "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" (UID: "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:19.812305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.812276 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-kube-api-access-ns5bm" pod "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" (UID: "41cf6621-92c3-49ea-b3d1-4b52ca5d88a7"). InnerVolumeSpecName "kube-api-access-ns5bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:19.909849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.909817 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.909849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.909847 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns5bm\" (UniqueName: \"kubernetes.io/projected/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-kube-api-access-ns5bm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.909849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.909857 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:19.910091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:19.909867 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:20.116549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.116458 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 01:22:20.117217 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.117180 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 01:22:20.366207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.366172 4927 generic.go:356] "Generic (PLEG): container finished" podID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerID="1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc" exitCode=0 Oct 30 01:22:20.366393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.366224 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" Oct 30 01:22:20.366393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.366261 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" event={"ID":"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7","Type":"ContainerDied","Data":"1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc"} Oct 30 01:22:20.366393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.366307 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6" event={"ID":"41cf6621-92c3-49ea-b3d1-4b52ca5d88a7","Type":"ContainerDied","Data":"e099ba88d70ba925a70053db61ad412aaba943240b49f32bf794a53d77def450"} Oct 30 01:22:20.366393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.366329 4927 scope.go:122] "RemoveContainer" containerID="1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc" Oct 30 01:22:20.404635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.404611 4927 scope.go:122] "RemoveContainer" containerID="1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc" Oct 30 01:22:20.404976 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:20.404952 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc\": container with ID starting with 1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc not found: ID does not exist" containerID="1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc" Oct 30 01:22:20.405061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.404983 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc"} err="failed to get container status \"1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc\": rpc error: code = NotFound desc = could not find container \"1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc\": container with ID starting with 1383935d797c65bde6bd2746ad6cd11d11420f9d4a35e000c782770e355ab3dc not found: ID does not exist" Oct 30 01:22:20.440595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.440548 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6"] Oct 30 01:22:20.444694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:20.444671 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-2-w4-6add35-kserve-75448b58bc-lrms6"] Oct 30 01:22:21.112705 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:21.112665 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 01:22:21.112883 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:21.112736 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 01:22:21.599289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:21.577167 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" path="/var/lib/kubelet/pods/41cf6621-92c3-49ea-b3d1-4b52ca5d88a7/volumes" Oct 30 01:22:22.113345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:22.113307 4927 patch_prober.go:29] interesting pod/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 01:22:22.113565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:22.113373 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 01:22:23.119560 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:23.119526 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:23.120061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:23.120039 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:23.120690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:23.120668 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:22:23.312856 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:23.312811 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" podUID="789da654-992f-48cc-80d2-5b069ed77feb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:22:23.313009 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:23.312936 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" podUID="bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:22:24.307852 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:24.307811 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:22:25.685712 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:25.685679 4927 scope.go:122] "RemoveContainer" containerID="ceb845f283448f3fbe55d9d4875d1935534057eda2ea77dd0ab6f3d3e03079fd" Oct 30 01:22:26.901093 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:26.901048 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" containerID="cri-o://1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3" gracePeriod=45 Oct 30 01:22:26.906713 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:26.906680 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" containerID="cri-o://7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5" gracePeriod=45 Oct 30 01:22:27.026637 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.026596 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": dial tcp 10.132.0.92:8000: connect: connection refused" Oct 30 01:22:27.260558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.260527 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 01:22:27.265865 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.265838 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 01:22:27.332501 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.332465 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-dshm\") pod \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " Oct 30 01:22:27.332689 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.332513 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598fbcd3-e89c-4308-833e-fa9fc2e3251c-kube-api-access-5gszj\" (UniqueName: \"kubernetes.io/projected/598fbcd3-e89c-4308-833e-fa9fc2e3251c-kube-api-access-5gszj\") pod \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " Oct 30 01:22:27.332689 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.332633 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-model-cache\") pod \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " Oct 30 01:22:27.332939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.332915 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-model-cache" pod "598fbcd3-e89c-4308-833e-fa9fc2e3251c" (UID: "598fbcd3-e89c-4308-833e-fa9fc2e3251c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333036 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tmp-dir\") pod \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333072 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-model-cache\") pod \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333225 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tmp-dir\") pod \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333261 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tls-certs\" (UniqueName: \"kubernetes.io/secret/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tls-certs\") pod \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333290 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tls-certs\") pod \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333321 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-dshm\") pod \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333389 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-home\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-home\") pod \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\" (UID: \"598fbcd3-e89c-4308-833e-fa9fc2e3251c\") " Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333423 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194f5c4e-881f-4fd2-b053-4d394f77cc5e-kube-api-access-dsk8g\" (UniqueName: \"kubernetes.io/projected/194f5c4e-881f-4fd2-b053-4d394f77cc5e-kube-api-access-dsk8g\") pod \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333501 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tmp-dir" pod "598fbcd3-e89c-4308-833e-fa9fc2e3251c" (UID: "598fbcd3-e89c-4308-833e-fa9fc2e3251c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333509 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-home\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-home\") pod \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\" (UID: \"194f5c4e-881f-4fd2-b053-4d394f77cc5e\") " Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333702 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-home" pod "194f5c4e-881f-4fd2-b053-4d394f77cc5e" (UID: "194f5c4e-881f-4fd2-b053-4d394f77cc5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.334017 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-model-cache" pod "194f5c4e-881f-4fd2-b053-4d394f77cc5e" (UID: "194f5c4e-881f-4fd2-b053-4d394f77cc5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.334235 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-home" pod "598fbcd3-e89c-4308-833e-fa9fc2e3251c" (UID: "598fbcd3-e89c-4308-833e-fa9fc2e3251c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.334556 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.334632 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.334647 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.334663 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.334678 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.334723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.333222 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tmp-dir" pod "194f5c4e-881f-4fd2-b053-4d394f77cc5e" (UID: "194f5c4e-881f-4fd2-b053-4d394f77cc5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.336489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.336460 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598fbcd3-e89c-4308-833e-fa9fc2e3251c-kube-api-access-5gszj" pod "598fbcd3-e89c-4308-833e-fa9fc2e3251c" (UID: "598fbcd3-e89c-4308-833e-fa9fc2e3251c"). InnerVolumeSpecName "kube-api-access-5gszj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:27.336666 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.336622 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-dshm" pod "194f5c4e-881f-4fd2-b053-4d394f77cc5e" (UID: "194f5c4e-881f-4fd2-b053-4d394f77cc5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.338366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.338337 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-dshm" pod "598fbcd3-e89c-4308-833e-fa9fc2e3251c" (UID: "598fbcd3-e89c-4308-833e-fa9fc2e3251c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:27.338502 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.338473 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194f5c4e-881f-4fd2-b053-4d394f77cc5e-kube-api-access-dsk8g" pod "194f5c4e-881f-4fd2-b053-4d394f77cc5e" (UID: "194f5c4e-881f-4fd2-b053-4d394f77cc5e"). InnerVolumeSpecName "kube-api-access-dsk8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:27.338900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.338875 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tls-certs" pod "194f5c4e-881f-4fd2-b053-4d394f77cc5e" (UID: "194f5c4e-881f-4fd2-b053-4d394f77cc5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:27.339141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.339117 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tls-certs" pod "598fbcd3-e89c-4308-833e-fa9fc2e3251c" (UID: "598fbcd3-e89c-4308-833e-fa9fc2e3251c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:27.436157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.436108 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598fbcd3-e89c-4308-833e-fa9fc2e3251c-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.436157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.436149 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.436157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.436160 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598fbcd3-e89c-4308-833e-fa9fc2e3251c-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.436433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.436175 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsk8g\" (UniqueName: \"kubernetes.io/projected/194f5c4e-881f-4fd2-b053-4d394f77cc5e-kube-api-access-dsk8g\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.436433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.436187 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.436433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.436198 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gszj\" (UniqueName: \"kubernetes.io/projected/598fbcd3-e89c-4308-833e-fa9fc2e3251c-kube-api-access-5gszj\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.436433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.436209 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194f5c4e-881f-4fd2-b053-4d394f77cc5e-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:27.512851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.512748 4927 generic.go:356] "Generic (PLEG): container finished" podID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerID="7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5" exitCode=0 Oct 30 01:22:27.512851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.512816 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" Oct 30 01:22:27.512851 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.512828 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" event={"ID":"194f5c4e-881f-4fd2-b053-4d394f77cc5e","Type":"ContainerDied","Data":"7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5"} Oct 30 01:22:27.513142 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.512882 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" event={"ID":"194f5c4e-881f-4fd2-b053-4d394f77cc5e","Type":"ContainerDied","Data":"8093871dccb1cc97491389d588a8fbe3877c6fcf7ee1bb868f6a62e1ef896adf"} Oct 30 01:22:27.513142 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.512906 4927 scope.go:122] "RemoveContainer" containerID="7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5" Oct 30 01:22:27.514835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.514813 4927 generic.go:356] "Generic (PLEG): container finished" podID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerID="1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3" exitCode=0 Oct 30 01:22:27.514956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.514920 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" Oct 30 01:22:27.514956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.514947 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" event={"ID":"598fbcd3-e89c-4308-833e-fa9fc2e3251c","Type":"ContainerDied","Data":"1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3"} Oct 30 01:22:27.515087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.514970 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s" event={"ID":"598fbcd3-e89c-4308-833e-fa9fc2e3251c","Type":"ContainerDied","Data":"195333d0a98577babe21e1312bc7eaa615fa168a5159e1fd77a6b72f0a3c1b0b"} Oct 30 01:22:27.529492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.528799 4927 scope.go:122] "RemoveContainer" containerID="7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5" Oct 30 01:22:27.529492 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:27.529138 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5\": container with ID starting with 7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5 not found: ID does not exist" containerID="7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5" Oct 30 01:22:27.529492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.529164 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5"} err="failed to get container status \"7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5\": rpc error: code = NotFound desc = could not find container \"7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5\": container with ID starting with 7237a3834b57db5f1f9f2efcacafd1a5be6ed18c6b018fb2e99f08ff2b384ee5 not found: ID does not exist" Oct 30 01:22:27.529492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.529182 4927 scope.go:122] "RemoveContainer" containerID="1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3" Oct 30 01:22:27.543138 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.543109 4927 scope.go:122] "RemoveContainer" containerID="1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3" Oct 30 01:22:27.543503 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:27.543482 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3\": container with ID starting with 1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3 not found: ID does not exist" containerID="1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3" Oct 30 01:22:27.543588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.543514 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3"} err="failed to get container status \"1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3\": rpc error: code = NotFound desc = could not find container \"1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3\": container with ID starting with 1761a76932c8c738229d7f4abd084cd786b540c2917342567ccbbb7a96e253f3 not found: ID does not exist" Oct 30 01:22:27.578806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.578767 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v"] Oct 30 01:22:27.584402 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.584374 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v"] Oct 30 01:22:27.616432 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.616396 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s"] Oct 30 01:22:27.620375 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:27.620352 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/distinct-2-w0-2b4105-kserve-5858c844f8-m2m2s"] Oct 30 01:22:28.136161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:28.136119 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/distinct-w0-2b4105-kserve-bc54c9645-78x5v" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:22:29.573230 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:29.573190 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" path="/var/lib/kubelet/pods/194f5c4e-881f-4fd2-b053-4d394f77cc5e/volumes" Oct 30 01:22:29.573715 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:29.573700 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" path="/var/lib/kubelet/pods/598fbcd3-e89c-4308-833e-fa9fc2e3251c/volumes" Oct 30 01:22:33.312818 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:33.312640 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" podUID="789da654-992f-48cc-80d2-5b069ed77feb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:22:33.312818 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:33.312808 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" podUID="bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:22:34.444304 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:34.444267 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:22:35.579371 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.579338 4927 generic.go:356] "Generic (PLEG): container finished" podID="789da654-992f-48cc-80d2-5b069ed77feb" containerID="eae39647c935f34c51c0be9ece233978850257cec10d101966142e1303efda9a" exitCode=0 Oct 30 01:22:35.579803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.579412 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" event={"ID":"789da654-992f-48cc-80d2-5b069ed77feb","Type":"ContainerDied","Data":"eae39647c935f34c51c0be9ece233978850257cec10d101966142e1303efda9a"} Oct 30 01:22:35.788301 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.788276 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 01:22:35.834214 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.834166 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 01:22:35.851849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.851815 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789da654-992f-48cc-80d2-5b069ed77feb-kube-api-access-f4m56\" (UniqueName: \"kubernetes.io/projected/789da654-992f-48cc-80d2-5b069ed77feb-kube-api-access-f4m56\") pod \"789da654-992f-48cc-80d2-5b069ed77feb\" (UID: \"789da654-992f-48cc-80d2-5b069ed77feb\") " Oct 30 01:22:35.852033 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.851873 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789da654-992f-48cc-80d2-5b069ed77feb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da654-992f-48cc-80d2-5b069ed77feb-plugins-config-volume\") pod \"789da654-992f-48cc-80d2-5b069ed77feb\" (UID: \"789da654-992f-48cc-80d2-5b069ed77feb\") " Oct 30 01:22:35.852363 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.852313 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789da654-992f-48cc-80d2-5b069ed77feb-plugins-config-volume" pod "789da654-992f-48cc-80d2-5b069ed77feb" (UID: "789da654-992f-48cc-80d2-5b069ed77feb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:35.852518 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.852496 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da654-992f-48cc-80d2-5b069ed77feb-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:35.854196 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.854171 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789da654-992f-48cc-80d2-5b069ed77feb-kube-api-access-f4m56" pod "789da654-992f-48cc-80d2-5b069ed77feb" (UID: "789da654-992f-48cc-80d2-5b069ed77feb"). InnerVolumeSpecName "kube-api-access-f4m56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:35.953161 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.953122 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-plugins-config-volume\") pod \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\" (UID: \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\") " Oct 30 01:22:35.953351 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.953230 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-kube-api-access-wm8cg\" (UniqueName: \"kubernetes.io/projected/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-kube-api-access-wm8cg\") pod \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\" (UID: \"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa\") " Oct 30 01:22:35.953530 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.953505 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-plugins-config-volume" pod "bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" (UID: "bb7e9d02-c97b-4bca-8bd7-11a746ccbafa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:35.953784 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.953768 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:35.953840 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.953787 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4m56\" (UniqueName: \"kubernetes.io/projected/789da654-992f-48cc-80d2-5b069ed77feb-kube-api-access-f4m56\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:35.955382 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:35.955362 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-kube-api-access-wm8cg" pod "bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" (UID: "bb7e9d02-c97b-4bca-8bd7-11a746ccbafa"). InnerVolumeSpecName "kube-api-access-wm8cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:36.055538 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.055454 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm8cg\" (UniqueName: \"kubernetes.io/projected/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa-kube-api-access-wm8cg\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:36.589376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.589345 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" Oct 30 01:22:36.589891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.589385 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd" event={"ID":"789da654-992f-48cc-80d2-5b069ed77feb","Type":"ContainerDied","Data":"12248254692d72e520a21a63b1464a914fbd6c3fe16c4ca9280ac4f717fb2118"} Oct 30 01:22:36.589891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.589445 4927 scope.go:122] "RemoveContainer" containerID="eae39647c935f34c51c0be9ece233978850257cec10d101966142e1303efda9a" Oct 30 01:22:36.593079 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.592327 4927 generic.go:356] "Generic (PLEG): container finished" podID="bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" containerID="68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66" exitCode=0 Oct 30 01:22:36.593079 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.592404 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" event={"ID":"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa","Type":"ContainerDied","Data":"68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66"} Oct 30 01:22:36.593079 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.592497 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" event={"ID":"bb7e9d02-c97b-4bca-8bd7-11a746ccbafa","Type":"ContainerDied","Data":"4ac996b700cef866cdf4f6b6b2cc9d60a1e11d18d01762f65f0d7dfd20bcfaec"} Oct 30 01:22:36.593079 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.592613 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86" Oct 30 01:22:36.625785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.625745 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4"] Oct 30 01:22:36.628424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.626077 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" podUID="1cfdc6a3-6c99-405e-a071-1ee80c4e6911" containerName="maas-api" containerID="cri-o://da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6" gracePeriod=30 Oct 30 01:22:36.637529 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.637470 4927 scope.go:122] "RemoveContainer" containerID="68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66" Oct 30 01:22:36.717878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.717851 4927 scope.go:122] "RemoveContainer" containerID="68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66" Oct 30 01:22:36.718230 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:36.718199 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66\": container with ID starting with 68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66 not found: ID does not exist" containerID="68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66" Oct 30 01:22:36.718321 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.718241 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66"} err="failed to get container status \"68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66\": rpc error: code = NotFound desc = could not find container \"68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66\": container with ID starting with 68093cacea0a7236fc5a4584c97e4ae4966bbdf3e7672d48e09fe4b579f46f66 not found: ID does not exist" Oct 30 01:22:36.737671 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.737638 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86"] Oct 30 01:22:36.745667 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.745641 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6add35-b8dcb9999-mhp86"] Oct 30 01:22:36.780594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.777112 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd"] Oct 30 01:22:36.784093 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.782844 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6add35-7cd86547c6-gnkgd"] Oct 30 01:22:36.866754 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.866713 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c"] Oct 30 01:22:36.867034 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.867004 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" podUID="47739eb1-7fec-48dc-91c7-405c4ecc53c8" containerName="payload-processing" containerID="cri-o://cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139" gracePeriod=30 Oct 30 01:22:36.903618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.900121 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48"] Oct 30 01:22:36.903618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:36.900406 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" podUID="60e24a86-fcf6-4532-b9fd-7dc8b33d374e" containerName="payload-pre-processing" containerID="cri-o://e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab" gracePeriod=30 Oct 30 01:22:37.042033 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.042004 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:22:37.172283 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.172243 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-metrics-certs\") pod \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " Oct 30 01:22:37.172474 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.172330 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls\") pod \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " Oct 30 01:22:37.172474 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.172461 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-openshift-service-ca\") pod \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " Oct 30 01:22:37.172634 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.172560 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-kube-api-access-6wcvz\" (UniqueName: \"kubernetes.io/projected/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-kube-api-access-6wcvz\") pod \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\" (UID: \"1cfdc6a3-6c99-405e-a071-1ee80c4e6911\") " Oct 30 01:22:37.174647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.174612 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-openshift-service-ca" pod "1cfdc6a3-6c99-405e-a071-1ee80c4e6911" (UID: "1cfdc6a3-6c99-405e-a071-1ee80c4e6911"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:37.177512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.177483 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-kube-api-access-6wcvz" pod "1cfdc6a3-6c99-405e-a071-1ee80c4e6911" (UID: "1cfdc6a3-6c99-405e-a071-1ee80c4e6911"). InnerVolumeSpecName "kube-api-access-6wcvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:37.178799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.178772 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls" pod "1cfdc6a3-6c99-405e-a071-1ee80c4e6911" (UID: "1cfdc6a3-6c99-405e-a071-1ee80c4e6911"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:37.180250 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.180224 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-metrics-certs" pod "1cfdc6a3-6c99-405e-a071-1ee80c4e6911" (UID: "1cfdc6a3-6c99-405e-a071-1ee80c4e6911"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:37.324688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.285747 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:37.324688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.285788 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wcvz\" (UniqueName: \"kubernetes.io/projected/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-kube-api-access-6wcvz\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:37.324688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.285803 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:37.324688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.285818 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfdc6a3-6c99-405e-a071-1ee80c4e6911-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:37.343640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.343545 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:37.358288 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.358262 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:37.387265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.387213 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-kube-api-access-dxst2\" (UniqueName: \"kubernetes.io/projected/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-kube-api-access-dxst2\") pod \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\" (UID: \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\") " Oct 30 01:22:37.387468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.387278 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47739eb1-7fec-48dc-91c7-405c4ecc53c8-kube-api-access-2xdvb\" (UniqueName: \"kubernetes.io/projected/47739eb1-7fec-48dc-91c7-405c4ecc53c8-kube-api-access-2xdvb\") pod \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\" (UID: \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\") " Oct 30 01:22:37.387468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.387319 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-plugins-config-volume\") pod \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\" (UID: \"60e24a86-fcf6-4532-b9fd-7dc8b33d374e\") " Oct 30 01:22:37.387468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.387351 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47739eb1-7fec-48dc-91c7-405c4ecc53c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47739eb1-7fec-48dc-91c7-405c4ecc53c8-plugins-config-volume\") pod \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\" (UID: \"47739eb1-7fec-48dc-91c7-405c4ecc53c8\") " Oct 30 01:22:37.388290 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.388259 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47739eb1-7fec-48dc-91c7-405c4ecc53c8-plugins-config-volume" pod "47739eb1-7fec-48dc-91c7-405c4ecc53c8" (UID: "47739eb1-7fec-48dc-91c7-405c4ecc53c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:37.388438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.388245 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-plugins-config-volume" pod "60e24a86-fcf6-4532-b9fd-7dc8b33d374e" (UID: "60e24a86-fcf6-4532-b9fd-7dc8b33d374e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:37.394614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.390599 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-kube-api-access-dxst2" pod "60e24a86-fcf6-4532-b9fd-7dc8b33d374e" (UID: "60e24a86-fcf6-4532-b9fd-7dc8b33d374e"). InnerVolumeSpecName "kube-api-access-dxst2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:37.394614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.391663 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47739eb1-7fec-48dc-91c7-405c4ecc53c8-kube-api-access-2xdvb" pod "47739eb1-7fec-48dc-91c7-405c4ecc53c8" (UID: "47739eb1-7fec-48dc-91c7-405c4ecc53c8"). InnerVolumeSpecName "kube-api-access-2xdvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:37.489504 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.489472 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxst2\" (UniqueName: \"kubernetes.io/projected/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-kube-api-access-dxst2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:37.489504 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.489504 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xdvb\" (UniqueName: \"kubernetes.io/projected/47739eb1-7fec-48dc-91c7-405c4ecc53c8-kube-api-access-2xdvb\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:37.489748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.489514 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e24a86-fcf6-4532-b9fd-7dc8b33d374e-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:37.489748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.489524 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47739eb1-7fec-48dc-91c7-405c4ecc53c8-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:22:37.572939 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.572906 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789da654-992f-48cc-80d2-5b069ed77feb" path="/var/lib/kubelet/pods/789da654-992f-48cc-80d2-5b069ed77feb/volumes" Oct 30 01:22:37.573372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.573358 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" path="/var/lib/kubelet/pods/bb7e9d02-c97b-4bca-8bd7-11a746ccbafa/volumes" Oct 30 01:22:37.602744 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.602705 4927 generic.go:356] "Generic (PLEG): container finished" podID="1cfdc6a3-6c99-405e-a071-1ee80c4e6911" containerID="da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6" exitCode=0 Oct 30 01:22:37.603193 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.602756 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" Oct 30 01:22:37.603193 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.602793 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" event={"ID":"1cfdc6a3-6c99-405e-a071-1ee80c4e6911","Type":"ContainerDied","Data":"da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6"} Oct 30 01:22:37.603193 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.602855 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4" event={"ID":"1cfdc6a3-6c99-405e-a071-1ee80c4e6911","Type":"ContainerDied","Data":"d1fb5d4fc3f7979ec3e1385f009c6f1560d9bee5cdc60a5514cd24d8ed14efd3"} Oct 30 01:22:37.603193 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.602885 4927 scope.go:122] "RemoveContainer" containerID="da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6" Oct 30 01:22:37.605806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.605782 4927 generic.go:356] "Generic (PLEG): container finished" podID="60e24a86-fcf6-4532-b9fd-7dc8b33d374e" containerID="e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab" exitCode=0 Oct 30 01:22:37.605934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.605865 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" event={"ID":"60e24a86-fcf6-4532-b9fd-7dc8b33d374e","Type":"ContainerDied","Data":"e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab"} Oct 30 01:22:37.605934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.605907 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" event={"ID":"60e24a86-fcf6-4532-b9fd-7dc8b33d374e","Type":"ContainerDied","Data":"1a02ad7b5dc50332318603d47d599deb449ec15ea899a9c792378643545b3a01"} Oct 30 01:22:37.605934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.605905 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48" Oct 30 01:22:37.608963 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.608942 4927 generic.go:356] "Generic (PLEG): container finished" podID="47739eb1-7fec-48dc-91c7-405c4ecc53c8" containerID="cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139" exitCode=0 Oct 30 01:22:37.609084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.608983 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" Oct 30 01:22:37.609084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.609008 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" event={"ID":"47739eb1-7fec-48dc-91c7-405c4ecc53c8","Type":"ContainerDied","Data":"cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139"} Oct 30 01:22:37.609084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.609037 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c" event={"ID":"47739eb1-7fec-48dc-91c7-405c4ecc53c8","Type":"ContainerDied","Data":"dc2fa3ee7d592e047977ddd5999c6cbf1711666b588f8f83e637a18db7433e24"} Oct 30 01:22:37.647918 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.647884 4927 scope.go:122] "RemoveContainer" containerID="da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6" Oct 30 01:22:37.648430 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:37.648394 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6\": container with ID starting with da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6 not found: ID does not exist" containerID="da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6" Oct 30 01:22:37.648592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.648435 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6"} err="failed to get container status \"da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6\": rpc error: code = NotFound desc = could not find container \"da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6\": container with ID starting with da1aba6b372b3abff6b4b8eab329a8bc3c63e40ee8d3907a37958868ab94ede6 not found: ID does not exist" Oct 30 01:22:37.648592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.648461 4927 scope.go:122] "RemoveContainer" containerID="e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab" Oct 30 01:22:37.674422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.674400 4927 scope.go:122] "RemoveContainer" containerID="e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab" Oct 30 01:22:37.674816 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:37.674786 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab\": container with ID starting with e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab not found: ID does not exist" containerID="e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab" Oct 30 01:22:37.674900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.674827 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab"} err="failed to get container status \"e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab\": rpc error: code = NotFound desc = could not find container \"e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab\": container with ID starting with e8d880aa390cbca11220bd64ea6a31a4cee3d18dddd4720efe39b952dd3083ab not found: ID does not exist" Oct 30 01:22:37.674900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.674851 4927 scope.go:122] "RemoveContainer" containerID="cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139" Oct 30 01:22:37.685705 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.685680 4927 scope.go:122] "RemoveContainer" containerID="cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139" Oct 30 01:22:37.686023 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:22:37.685994 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139\": container with ID starting with cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139 not found: ID does not exist" containerID="cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139" Oct 30 01:22:37.686078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.686036 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139"} err="failed to get container status \"cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139\": rpc error: code = NotFound desc = could not find container \"cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139\": container with ID starting with cbeb1d67d9ad3d8333c3276233b6fee6bca6344f70ffee0d313f6270bcc22139 not found: ID does not exist" Oct 30 01:22:37.696662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.696564 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4"] Oct 30 01:22:37.702945 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.702916 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6cba49-75d64cd74c-zh4c4"] Oct 30 01:22:37.729800 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.729772 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c"] Oct 30 01:22:37.734198 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.734168 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6cba49-9df45c854-hqw6c"] Oct 30 01:22:37.766354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.766315 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48"] Oct 30 01:22:37.772074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:37.772046 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6cba49-676cc5bbcf-tqk48"] Oct 30 01:22:39.573370 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:39.573336 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfdc6a3-6c99-405e-a071-1ee80c4e6911" path="/var/lib/kubelet/pods/1cfdc6a3-6c99-405e-a071-1ee80c4e6911/volumes" Oct 30 01:22:39.573882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:39.573864 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47739eb1-7fec-48dc-91c7-405c4ecc53c8" path="/var/lib/kubelet/pods/47739eb1-7fec-48dc-91c7-405c4ecc53c8/volumes" Oct 30 01:22:39.574281 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:39.574254 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e24a86-fcf6-4532-b9fd-7dc8b33d374e" path="/var/lib/kubelet/pods/60e24a86-fcf6-4532-b9fd-7dc8b33d374e/volumes" Oct 30 01:22:41.690012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.689974 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl"] Oct 30 01:22:41.691000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.690976 4927 cpu_manager.go:414] "containerMap: removing container" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" Oct 30 01:22:41.691000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.690999 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691013 4927 cpu_manager.go:414] "containerMap: removing container" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691021 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691044 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" containerName="payload-pre-processing" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691051 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" containerName="payload-pre-processing" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691067 4927 cpu_manager.go:414] "containerMap: removing container" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691074 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691115 4927 cpu_manager.go:414] "containerMap: removing container" podUID="789da654-992f-48cc-80d2-5b069ed77feb" containerName="payload-processing" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691122 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789da654-992f-48cc-80d2-5b069ed77feb" containerName="payload-processing" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691136 4927 cpu_manager.go:414] "containerMap: removing container" podUID="47739eb1-7fec-48dc-91c7-405c4ecc53c8" containerName="payload-processing" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691142 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47739eb1-7fec-48dc-91c7-405c4ecc53c8" containerName="payload-processing" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691153 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1cfdc6a3-6c99-405e-a071-1ee80c4e6911" containerName="maas-api" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691160 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfdc6a3-6c99-405e-a071-1ee80c4e6911" containerName="maas-api" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691192 4927 cpu_manager.go:414] "containerMap: removing container" podUID="60e24a86-fcf6-4532-b9fd-7dc8b33d374e" containerName="payload-pre-processing" Oct 30 01:22:41.691220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691200 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e24a86-fcf6-4532-b9fd-7dc8b33d374e" containerName="payload-pre-processing" Oct 30 01:22:41.691972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691390 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e24a86-fcf6-4532-b9fd-7dc8b33d374e" containerName="payload-pre-processing" Oct 30 01:22:41.691972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691409 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="194f5c4e-881f-4fd2-b053-4d394f77cc5e" containerName="main" Oct 30 01:22:41.691972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691424 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="598fbcd3-e89c-4308-833e-fa9fc2e3251c" containerName="main" Oct 30 01:22:41.691972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691435 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7e9d02-c97b-4bca-8bd7-11a746ccbafa" containerName="payload-pre-processing" Oct 30 01:22:41.691972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691444 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cf6621-92c3-49ea-b3d1-4b52ca5d88a7" containerName="main" Oct 30 01:22:41.691972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691458 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="789da654-992f-48cc-80d2-5b069ed77feb" containerName="payload-processing" Oct 30 01:22:41.691972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691467 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="47739eb1-7fec-48dc-91c7-405c4ecc53c8" containerName="payload-processing" Oct 30 01:22:41.691972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.691477 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfdc6a3-6c99-405e-a071-1ee80c4e6911" containerName="maas-api" Oct 30 01:22:41.697051 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.697025 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.727378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.713498 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3e9921-serving-cert\"" Oct 30 01:22:41.727378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.724010 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl"] Oct 30 01:22:41.757802 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.757763 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.758002 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.757834 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-metrics-certs\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.758002 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.757960 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jq2g\" (UniqueName: \"kubernetes.io/projected/febc6e30-896a-495a-af9c-7a6a65e40975-kube-api-access-9jq2g\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.758130 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.758081 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febc6e30-896a-495a-af9c-7a6a65e40975-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.859557 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.859514 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febc6e30-896a-495a-af9c-7a6a65e40975-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.859790 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.859765 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.859844 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.859827 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-metrics-certs\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.859921 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.859903 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jq2g\" (UniqueName: \"kubernetes.io/projected/febc6e30-896a-495a-af9c-7a6a65e40975-kube-api-access-9jq2g\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.860215 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.860195 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febc6e30-896a-495a-af9c-7a6a65e40975-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.862541 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.862508 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-metrics-certs\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.862701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.862508 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:41.885000 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:41.884968 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jq2g\" (UniqueName: \"kubernetes.io/projected/febc6e30-896a-495a-af9c-7a6a65e40975-kube-api-access-9jq2g\") pod \"maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:42.016329 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:42.016244 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:42.460510 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:42.457231 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl"] Oct 30 01:22:42.460796 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:22:42.460201 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebc6e30_896a_495a_af9c_7a6a65e40975.slice/crio-7cedf422dafd9718d471eda0c85b60f908e787b1c56e71d376979721363dc26c WatchSource:0}: Error finding container 7cedf422dafd9718d471eda0c85b60f908e787b1c56e71d376979721363dc26c: Status 404 returned error can't find the container with id 7cedf422dafd9718d471eda0c85b60f908e787b1c56e71d376979721363dc26c Oct 30 01:22:42.675978 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:42.675931 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" event={"ID":"febc6e30-896a-495a-af9c-7a6a65e40975","Type":"ContainerStarted","Data":"7cedf422dafd9718d471eda0c85b60f908e787b1c56e71d376979721363dc26c"} Oct 30 01:22:43.686878 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:43.686830 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" event={"ID":"febc6e30-896a-495a-af9c-7a6a65e40975","Type":"ContainerStarted","Data":"f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270"} Oct 30 01:22:43.687457 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:43.686921 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:43.730717 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:43.730664 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" podStartSLOduration=2.31538756 podStartE2EDuration="2.730645934s" totalImagesPullingTime="415.258374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:43.729319024 +0000 UTC m=+2902.301245242" watchObservedRunningTime="2026-10-30 01:22:43.730645934 +0000 UTC m=+2902.302572155" Oct 30 01:22:49.700008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:49.699977 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:22:56.106227 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:56.105933 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b"] Oct 30 01:22:56.139457 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:22:56.139426 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7"] Oct 30 01:23:01.107199 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:01.107147 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" podUID="5725f313-f8ee-40cd-9ef8-f17e18d8ab60" containerName="payload-processing" containerID="cri-o://163076a131f8a6395fc1e891f2b01cc5f9666547e61e0f2ab21eef5c5d2a36f2" gracePeriod=25 Oct 30 01:23:01.140338 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:01.140287 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" podUID="e22f4c73-42a9-41b9-8f86-2dd9b4d41779" containerName="payload-pre-processing" containerID="cri-o://d6f0d3583acd48fa1d588fa8b7dfc7aabc890847815dbf2c887e40e26c483107" gracePeriod=25 Oct 30 01:23:04.443246 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:04.443193 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:23:05.250069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:05.250010 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b"] Oct 30 01:23:05.250713 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:05.250666 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" containerID="cri-o://36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946" gracePeriod=30 Oct 30 01:23:05.880064 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:05.880030 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7_23d22d03-5652-4aa6-a40c-6de31a390844/payload-processing/0.log" Oct 30 01:23:08.048008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.047913 4927 generic.go:356] "Generic (PLEG): container finished" podID="4836cefc-d5ca-469d-899b-d9298b44db87" containerID="3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426" exitCode=0 Oct 30 01:23:08.048008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.047986 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" event={"ID":"4836cefc-d5ca-469d-899b-d9298b44db87","Type":"ContainerDied","Data":"3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426"} Oct 30 01:23:08.048474 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.048459 4927 scope.go:122] "RemoveContainer" containerID="3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426" Oct 30 01:23:08.050273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.050243 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:23:08.050378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.050314 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:23:08.050378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.050350 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:23:08.496146 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.496112 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6"] Oct 30 01:23:08.496392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.496362 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" podUID="3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" containerName="maas-api" containerID="cri-o://f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d" gracePeriod=30 Oct 30 01:23:08.503605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.503536 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" podUID="3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.75:8443/health\": dial tcp 10.132.0.75:8443: connect: connection refused" Oct 30 01:23:08.868635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:08.867771 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 01:23:09.025822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.025044 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-kube-api-access-dw49m\" (UniqueName: \"kubernetes.io/projected/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-kube-api-access-dw49m\") pod \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " Oct 30 01:23:09.025822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.025088 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-metrics-certs\") pod \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " Oct 30 01:23:09.025822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.025136 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-openshift-service-ca\") pod \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " Oct 30 01:23:09.025822 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.025221 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-maas-api-tls\") pod \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\" (UID: \"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683\") " Oct 30 01:23:09.026804 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.026763 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-openshift-service-ca" pod "3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" (UID: "3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:09.028854 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.028819 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-kube-api-access-dw49m" pod "3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" (UID: "3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683"). InnerVolumeSpecName "kube-api-access-dw49m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:09.028985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.028926 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-maas-api-tls" pod "3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" (UID: "3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:09.029054 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.029032 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-metrics-certs" pod "3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" (UID: "3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:09.063833 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.063796 4927 generic.go:356] "Generic (PLEG): container finished" podID="3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" containerID="f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d" exitCode=0 Oct 30 01:23:09.064367 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.063884 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" Oct 30 01:23:09.064367 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.063884 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" event={"ID":"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683","Type":"ContainerDied","Data":"f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d"} Oct 30 01:23:09.064367 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.063955 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6" event={"ID":"3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683","Type":"ContainerDied","Data":"748a2b2fe84324fb9baee31d00ea16591c04e55912c5c31e93d5a2e603b523f3"} Oct 30 01:23:09.064367 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.063987 4927 scope.go:122] "RemoveContainer" containerID="f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d" Oct 30 01:23:09.066894 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.066825 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" event={"ID":"4836cefc-d5ca-469d-899b-d9298b44db87","Type":"ContainerStarted","Data":"79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1"} Oct 30 01:23:09.076763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.076744 4927 scope.go:122] "RemoveContainer" containerID="f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d" Oct 30 01:23:09.077046 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:09.077023 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d\": container with ID starting with f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d not found: ID does not exist" containerID="f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d" Oct 30 01:23:09.077172 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.077050 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d"} err="failed to get container status \"f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d\": rpc error: code = NotFound desc = could not find container \"f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d\": container with ID starting with f0fa316fcccd1c78ddefc074ee8d88ec90bae2c8a22bde3790241402d063551d not found: ID does not exist" Oct 30 01:23:09.116281 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.116205 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6"] Oct 30 01:23:09.121222 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.121192 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2b4105-c6c5c9bbd-nc5x6"] Oct 30 01:23:09.126694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.126668 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw49m\" (UniqueName: \"kubernetes.io/projected/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-kube-api-access-dw49m\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.126790 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.126722 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.126967 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.126947 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.127016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.126973 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.204462 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.204059 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:23:09.205818 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.205783 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:23:09.205997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.205840 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:23:09.208280 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.208251 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w"] Oct 30 01:23:09.209070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.209049 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" containerName="maas-api" Oct 30 01:23:09.209178 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.209070 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" containerName="maas-api" Oct 30 01:23:09.209273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.209258 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" containerName="maas-api" Oct 30 01:23:09.214508 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.214487 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.228177 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.228150 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.228339 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.228196 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.228339 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.228294 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.228519 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.228345 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.228519 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.228418 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.228692 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.228514 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.228692 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.228655 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.228786 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.228766 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njft\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-kube-api-access-9njft\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.228858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.228838 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d82372c-ad61-4079-9412-71ae61bf9c0d-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.236466 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.235791 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2b4105-openshift-default-dockercfg-wzpvv\"" Oct 30 01:23:09.239135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.239115 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w"] Oct 30 01:23:09.263633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.251060 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.330786 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.330820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.330853 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.332153 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.332454 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.330925 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.332784 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njft\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-kube-api-access-9njft\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.332823 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d82372c-ad61-4079-9412-71ae61bf9c0d-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.332930 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.332964 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.333024 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.333374 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.335079 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d82372c-ad61-4079-9412-71ae61bf9c0d-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.338938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.335959 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.356743 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.350464 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.366905 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.366851 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.369203 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.369143 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njft\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-kube-api-access-9njft\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.370347 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.370322 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.529963 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.529926 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:09.572533 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.572495 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683" path="/var/lib/kubelet/pods/3c7e2e89-9ed5-4ec1-96e6-a37f5cff6683/volumes" Oct 30 01:23:09.745759 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.745725 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" podUID="5725f313-f8ee-40cd-9ef8-f17e18d8ab60" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:09.745944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.745729 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" podUID="e22f4c73-42a9-41b9-8f86-2dd9b4d41779" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:09.906796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.906762 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w"] Oct 30 01:23:09.908858 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:23:09.908827 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d82372c_ad61_4079_9412_71ae61bf9c0d.slice/crio-b7c946636476715933d082ca9bf54335de9993cace43aad1510748d6611c7068 WatchSource:0}: Error finding container b7c946636476715933d082ca9bf54335de9993cace43aad1510748d6611c7068: Status 404 returned error can't find the container with id b7c946636476715933d082ca9bf54335de9993cace43aad1510748d6611c7068 Oct 30 01:23:09.915145 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.915099 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:23:09.921587 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.915480 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:23:09.921587 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:09.915638 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:23:10.118419 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.117688 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" event={"ID":"2d82372c-ad61-4079-9412-71ae61bf9c0d","Type":"ContainerStarted","Data":"618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2"} Oct 30 01:23:10.118419 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.117780 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" event={"ID":"2d82372c-ad61-4079-9412-71ae61bf9c0d","Type":"ContainerStarted","Data":"b7c946636476715933d082ca9bf54335de9993cace43aad1510748d6611c7068"} Oct 30 01:23:10.162748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.162642 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" podStartSLOduration=1.16262771 podStartE2EDuration="1.16262771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:10.160965554 +0000 UTC m=+2928.732891777" watchObservedRunningTime="2026-10-30 01:23:10.16262771 +0000 UTC m=+2928.734553932" Oct 30 01:23:10.204346 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.204300 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:23:10.204675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.204629 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:23:10.547906 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.530916 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:10.662897 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.662861 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:23:10.754748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.754662 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-data\") pod \"9f4378ab-80c5-42d8-a817-8c11650d094b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " Oct 30 01:23:10.755114 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.755058 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-envoy\") pod \"9f4378ab-80c5-42d8-a817-8c11650d094b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " Oct 30 01:23:10.755341 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.755307 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4378ab-80c5-42d8-a817-8c11650d094b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4378ab-80c5-42d8-a817-8c11650d094b-istiod-ca-cert\") pod \"9f4378ab-80c5-42d8-a817-8c11650d094b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " Oct 30 01:23:10.755531 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.755512 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-certs\") pod \"9f4378ab-80c5-42d8-a817-8c11650d094b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " Oct 30 01:23:10.755730 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.755712 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-token\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-token\") pod \"9f4378ab-80c5-42d8-a817-8c11650d094b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " Oct 30 01:23:10.755935 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.755917 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-kube-api-access-8x5sq\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-kube-api-access-8x5sq\") pod \"9f4378ab-80c5-42d8-a817-8c11650d094b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " Oct 30 01:23:10.756086 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.756070 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-credential-socket\") pod \"9f4378ab-80c5-42d8-a817-8c11650d094b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " Oct 30 01:23:10.756240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.756224 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-socket\") pod \"9f4378ab-80c5-42d8-a817-8c11650d094b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " Oct 30 01:23:10.756403 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.756386 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-podinfo\") pod \"9f4378ab-80c5-42d8-a817-8c11650d094b\" (UID: \"9f4378ab-80c5-42d8-a817-8c11650d094b\") " Oct 30 01:23:10.757365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.757337 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-certs" pod "9f4378ab-80c5-42d8-a817-8c11650d094b" (UID: "9f4378ab-80c5-42d8-a817-8c11650d094b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:10.757469 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.757411 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4378ab-80c5-42d8-a817-8c11650d094b-istiod-ca-cert" pod "9f4378ab-80c5-42d8-a817-8c11650d094b" (UID: "9f4378ab-80c5-42d8-a817-8c11650d094b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:10.757630 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.757603 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-credential-socket" pod "9f4378ab-80c5-42d8-a817-8c11650d094b" (UID: "9f4378ab-80c5-42d8-a817-8c11650d094b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:10.757754 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.757730 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-data" pod "9f4378ab-80c5-42d8-a817-8c11650d094b" (UID: "9f4378ab-80c5-42d8-a817-8c11650d094b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:10.757861 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.757843 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-socket" pod "9f4378ab-80c5-42d8-a817-8c11650d094b" (UID: "9f4378ab-80c5-42d8-a817-8c11650d094b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:10.759741 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.759694 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-podinfo" pod "9f4378ab-80c5-42d8-a817-8c11650d094b" (UID: "9f4378ab-80c5-42d8-a817-8c11650d094b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:10.760339 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.760289 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-envoy" pod "9f4378ab-80c5-42d8-a817-8c11650d094b" (UID: "9f4378ab-80c5-42d8-a817-8c11650d094b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:10.761947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.761925 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-kube-api-access-8x5sq" pod "9f4378ab-80c5-42d8-a817-8c11650d094b" (UID: "9f4378ab-80c5-42d8-a817-8c11650d094b"). InnerVolumeSpecName "kube-api-access-8x5sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:10.762116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.762091 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-token" pod "9f4378ab-80c5-42d8-a817-8c11650d094b" (UID: "9f4378ab-80c5-42d8-a817-8c11650d094b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:10.858019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.857990 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.858019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.858016 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x5sq\" (UniqueName: \"kubernetes.io/projected/9f4378ab-80c5-42d8-a817-8c11650d094b-kube-api-access-8x5sq\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.858207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.858030 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.858207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.858045 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.858207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.858054 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.858207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.858062 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.858207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.858070 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.858207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.858077 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4378ab-80c5-42d8-a817-8c11650d094b-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.858207 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:10.858085 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4378ab-80c5-42d8-a817-8c11650d094b-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.138699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.138664 4927 generic.go:356] "Generic (PLEG): container finished" podID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerID="36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946" exitCode=0 Oct 30 01:23:11.139160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.138737 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" event={"ID":"9f4378ab-80c5-42d8-a817-8c11650d094b","Type":"ContainerDied","Data":"36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946"} Oct 30 01:23:11.139160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.138797 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" event={"ID":"9f4378ab-80c5-42d8-a817-8c11650d094b","Type":"ContainerDied","Data":"747ea1e79abfd5cb0323add57102682aa9ddd76cd636172e550818b67216ccea"} Oct 30 01:23:11.139160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.138821 4927 scope.go:122] "RemoveContainer" containerID="36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946" Oct 30 01:23:11.139160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.139096 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b" Oct 30 01:23:11.151002 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.150984 4927 scope.go:122] "RemoveContainer" containerID="36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946" Oct 30 01:23:11.151294 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:11.151273 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946\": container with ID starting with 36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946 not found: ID does not exist" containerID="36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946" Oct 30 01:23:11.151374 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.151301 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946"} err="failed to get container status \"36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946\": rpc error: code = NotFound desc = could not find container \"36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946\": container with ID starting with 36ae6f24f6d3b0247b752f8efe6fb13fb2300d9a9f83c80df4d439994cb1e946 not found: ID does not exist" Oct 30 01:23:11.209596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.208927 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:23:11.209596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.209004 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:23:11.227730 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.227679 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b"] Oct 30 01:23:11.232700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.232665 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-2fw7b"] Oct 30 01:23:11.532676 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.532258 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:23:11.532676 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.532327 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 01:23:11.585597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:11.574362 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" path="/var/lib/kubelet/pods/9f4378ab-80c5-42d8-a817-8c11650d094b/volumes" Oct 30 01:23:12.210594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:12.204985 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:23:12.210594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:12.205053 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:23:12.531350 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:12.531027 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:23:12.531350 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:12.531096 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 01:23:13.211036 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:13.210994 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:23:13.211701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:13.211665 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:23:13.534668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:13.534298 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:23:13.534668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:13.534389 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 01:23:14.225478 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:14.225126 4927 patch_prober.go:29] interesting pod/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 01:23:14.225478 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:14.225188 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 01:23:14.531110 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:14.531071 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:23:14.531359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:14.531129 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 01:23:15.209493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:15.209456 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:23:15.210771 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:15.210744 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:23:15.530649 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:15.530527 4927 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:23:15.530649 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:15.530621 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 01:23:16.334733 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:16.334686 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w"] Oct 30 01:23:16.335147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:16.335112 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerName="istio-proxy" containerID="cri-o://618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2" gracePeriod=30 Oct 30 01:23:17.398995 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:17.398944 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x"] Oct 30 01:23:18.098342 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.098302 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj"] Oct 30 01:23:18.098914 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.098877 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" podUID="f9676b8e-5fb3-438d-8138-b6f906d3dd52" containerName="maas-api" containerID="cri-o://4ac6609dbab63ead464e209be9cbfb402e8652ddc80314c0f74cd5036d11e86a" gracePeriod=30 Oct 30 01:23:18.378651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.376051 4927 generic.go:356] "Generic (PLEG): container finished" podID="f9676b8e-5fb3-438d-8138-b6f906d3dd52" containerID="4ac6609dbab63ead464e209be9cbfb402e8652ddc80314c0f74cd5036d11e86a" exitCode=0 Oct 30 01:23:18.378651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.376367 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" event={"ID":"f9676b8e-5fb3-438d-8138-b6f906d3dd52","Type":"ContainerDied","Data":"4ac6609dbab63ead464e209be9cbfb402e8652ddc80314c0f74cd5036d11e86a"} Oct 30 01:23:18.481804 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.481776 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 01:23:18.544409 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.544353 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-metrics-certs\") pod \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " Oct 30 01:23:18.544643 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.544429 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-maas-api-tls\") pod \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " Oct 30 01:23:18.544724 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.544692 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9676b8e-5fb3-438d-8138-b6f906d3dd52-kube-api-access-njk5x\" (UniqueName: \"kubernetes.io/projected/f9676b8e-5fb3-438d-8138-b6f906d3dd52-kube-api-access-njk5x\") pod \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " Oct 30 01:23:18.544997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.544971 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9676b8e-5fb3-438d-8138-b6f906d3dd52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9676b8e-5fb3-438d-8138-b6f906d3dd52-openshift-service-ca\") pod \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\" (UID: \"f9676b8e-5fb3-438d-8138-b6f906d3dd52\") " Oct 30 01:23:18.545966 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.545934 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9676b8e-5fb3-438d-8138-b6f906d3dd52-openshift-service-ca" pod "f9676b8e-5fb3-438d-8138-b6f906d3dd52" (UID: "f9676b8e-5fb3-438d-8138-b6f906d3dd52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:18.547792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.547767 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-metrics-certs" pod "f9676b8e-5fb3-438d-8138-b6f906d3dd52" (UID: "f9676b8e-5fb3-438d-8138-b6f906d3dd52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:18.548675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.548646 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-maas-api-tls" pod "f9676b8e-5fb3-438d-8138-b6f906d3dd52" (UID: "f9676b8e-5fb3-438d-8138-b6f906d3dd52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:18.548821 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.548791 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9676b8e-5fb3-438d-8138-b6f906d3dd52-kube-api-access-njk5x" pod "f9676b8e-5fb3-438d-8138-b6f906d3dd52" (UID: "f9676b8e-5fb3-438d-8138-b6f906d3dd52"). InnerVolumeSpecName "kube-api-access-njk5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:18.688651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.670009 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njk5x\" (UniqueName: \"kubernetes.io/projected/f9676b8e-5fb3-438d-8138-b6f906d3dd52-kube-api-access-njk5x\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:18.688651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.670053 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9676b8e-5fb3-438d-8138-b6f906d3dd52-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:18.688651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.670069 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:18.688651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:18.670083 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9676b8e-5fb3-438d-8138-b6f906d3dd52-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:19.385751 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:19.385712 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" Oct 30 01:23:19.385913 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:19.385749 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj" event={"ID":"f9676b8e-5fb3-438d-8138-b6f906d3dd52","Type":"ContainerDied","Data":"db1abdbeaf9a26a3e8e1cba6936edfe8d6a1d703bf99d0671fdd88b077b39e03"} Oct 30 01:23:19.385913 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:19.385816 4927 scope.go:122] "RemoveContainer" containerID="4ac6609dbab63ead464e209be9cbfb402e8652ddc80314c0f74cd5036d11e86a" Oct 30 01:23:19.452021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:19.451986 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj"] Oct 30 01:23:19.456891 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:19.456867 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6add35-5d8fdbccf7-gnmxj"] Oct 30 01:23:19.573140 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:19.573106 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9676b8e-5fb3-438d-8138-b6f906d3dd52" path="/var/lib/kubelet/pods/f9676b8e-5fb3-438d-8138-b6f906d3dd52/volumes" Oct 30 01:23:19.746549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:19.746458 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" podUID="e22f4c73-42a9-41b9-8f86-2dd9b4d41779" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:19.746549 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:19.746473 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" podUID="5725f313-f8ee-40cd-9ef8-f17e18d8ab60" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:21.416879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.416840 4927 generic.go:356] "Generic (PLEG): container finished" podID="5725f313-f8ee-40cd-9ef8-f17e18d8ab60" containerID="163076a131f8a6395fc1e891f2b01cc5f9666547e61e0f2ab21eef5c5d2a36f2" exitCode=0 Oct 30 01:23:21.417369 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.416930 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" event={"ID":"5725f313-f8ee-40cd-9ef8-f17e18d8ab60","Type":"ContainerDied","Data":"163076a131f8a6395fc1e891f2b01cc5f9666547e61e0f2ab21eef5c5d2a36f2"} Oct 30 01:23:21.417369 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.416989 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" event={"ID":"5725f313-f8ee-40cd-9ef8-f17e18d8ab60","Type":"ContainerDied","Data":"61f0bcc933dbbf47e3a6bc788b9e1be937975d15042a0e5490138d32038a5cdf"} Oct 30 01:23:21.417369 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.417017 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61f0bcc933dbbf47e3a6bc788b9e1be937975d15042a0e5490138d32038a5cdf" Oct 30 01:23:21.419020 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.418993 4927 generic.go:356] "Generic (PLEG): container finished" podID="e22f4c73-42a9-41b9-8f86-2dd9b4d41779" containerID="d6f0d3583acd48fa1d588fa8b7dfc7aabc890847815dbf2c887e40e26c483107" exitCode=0 Oct 30 01:23:21.419160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.419071 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" event={"ID":"e22f4c73-42a9-41b9-8f86-2dd9b4d41779","Type":"ContainerDied","Data":"d6f0d3583acd48fa1d588fa8b7dfc7aabc890847815dbf2c887e40e26c483107"} Oct 30 01:23:21.430623 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.430600 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 01:23:21.486633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.486601 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 01:23:21.611662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.611630 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-plugins-config-volume\") pod \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\" (UID: \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\") " Oct 30 01:23:21.611841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.611686 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-plugins-config-volume\") pod \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\" (UID: \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\") " Oct 30 01:23:21.611841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.611773 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-kube-api-access-n2pz7\" (UniqueName: \"kubernetes.io/projected/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-kube-api-access-n2pz7\") pod \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\" (UID: \"5725f313-f8ee-40cd-9ef8-f17e18d8ab60\") " Oct 30 01:23:21.611841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.611823 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-kube-api-access-hfp66\" (UniqueName: \"kubernetes.io/projected/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-kube-api-access-hfp66\") pod \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\" (UID: \"e22f4c73-42a9-41b9-8f86-2dd9b4d41779\") " Oct 30 01:23:21.612175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.612133 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-plugins-config-volume" pod "e22f4c73-42a9-41b9-8f86-2dd9b4d41779" (UID: "e22f4c73-42a9-41b9-8f86-2dd9b4d41779"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:21.612328 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.612180 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-plugins-config-volume" pod "5725f313-f8ee-40cd-9ef8-f17e18d8ab60" (UID: "5725f313-f8ee-40cd-9ef8-f17e18d8ab60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:21.612403 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.612351 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.612403 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.612368 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.614116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.614093 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-kube-api-access-n2pz7" pod "5725f313-f8ee-40cd-9ef8-f17e18d8ab60" (UID: "5725f313-f8ee-40cd-9ef8-f17e18d8ab60"). InnerVolumeSpecName "kube-api-access-n2pz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:21.614215 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.614123 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-kube-api-access-hfp66" pod "e22f4c73-42a9-41b9-8f86-2dd9b4d41779" (UID: "e22f4c73-42a9-41b9-8f86-2dd9b4d41779"). InnerVolumeSpecName "kube-api-access-hfp66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:21.714444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.714315 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2pz7\" (UniqueName: \"kubernetes.io/projected/5725f313-f8ee-40cd-9ef8-f17e18d8ab60-kube-api-access-n2pz7\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:21.714444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:21.714349 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfp66\" (UniqueName: \"kubernetes.io/projected/e22f4c73-42a9-41b9-8f86-2dd9b4d41779-kube-api-access-hfp66\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:22.400611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.400487 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" podUID="ac0599b6-0434-4304-be95-553cd7dd0ba3" containerName="payload-pre-processing" containerID="cri-o://49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f" gracePeriod=25 Oct 30 01:23:22.431788 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.431761 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" Oct 30 01:23:22.432154 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.431807 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7" event={"ID":"e22f4c73-42a9-41b9-8f86-2dd9b4d41779","Type":"ContainerDied","Data":"9711991f5ef3e65a8b21d887a0271dc12a9cfdcc56f66feb4585e36024765974"} Oct 30 01:23:22.432154 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.431863 4927 scope.go:122] "RemoveContainer" containerID="d6f0d3583acd48fa1d588fa8b7dfc7aabc890847815dbf2c887e40e26c483107" Oct 30 01:23:22.432154 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.431951 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b" Oct 30 01:23:22.506116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.506074 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7"] Oct 30 01:23:22.511671 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.511638 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ea2547-f846cf7c5-8mwn7"] Oct 30 01:23:22.551484 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.551446 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b"] Oct 30 01:23:22.556955 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.556923 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ea2547-5fbb8b67b8-q6n4b"] Oct 30 01:23:22.712408 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.712368 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:23:22.726290 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.726254 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0599b6-0434-4304-be95-553cd7dd0ba3-kube-api-access-778b8\" (UniqueName: \"kubernetes.io/projected/ac0599b6-0434-4304-be95-553cd7dd0ba3-kube-api-access-778b8\") pod \"ac0599b6-0434-4304-be95-553cd7dd0ba3\" (UID: \"ac0599b6-0434-4304-be95-553cd7dd0ba3\") " Oct 30 01:23:22.726453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.726395 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0599b6-0434-4304-be95-553cd7dd0ba3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599b6-0434-4304-be95-553cd7dd0ba3-plugins-config-volume\") pod \"ac0599b6-0434-4304-be95-553cd7dd0ba3\" (UID: \"ac0599b6-0434-4304-be95-553cd7dd0ba3\") " Oct 30 01:23:22.726859 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.726827 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0599b6-0434-4304-be95-553cd7dd0ba3-plugins-config-volume" pod "ac0599b6-0434-4304-be95-553cd7dd0ba3" (UID: "ac0599b6-0434-4304-be95-553cd7dd0ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:22.727090 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.727070 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599b6-0434-4304-be95-553cd7dd0ba3-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:22.729146 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.729117 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0599b6-0434-4304-be95-553cd7dd0ba3-kube-api-access-778b8" pod "ac0599b6-0434-4304-be95-553cd7dd0ba3" (UID: "ac0599b6-0434-4304-be95-553cd7dd0ba3"). InnerVolumeSpecName "kube-api-access-778b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:22.828147 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:22.828108 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-778b8\" (UniqueName: \"kubernetes.io/projected/ac0599b6-0434-4304-be95-553cd7dd0ba3-kube-api-access-778b8\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.343715 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.343684 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:23.436665 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.436601 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-kube-api-access-9njft\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-kube-api-access-9njft\") pod \"2d82372c-ad61-4079-9412-71ae61bf9c0d\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " Oct 30 01:23:23.437323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.436710 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-certs\") pod \"2d82372c-ad61-4079-9412-71ae61bf9c0d\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " Oct 30 01:23:23.437323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.436781 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d82372c-ad61-4079-9412-71ae61bf9c0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d82372c-ad61-4079-9412-71ae61bf9c0d-istiod-ca-cert\") pod \"2d82372c-ad61-4079-9412-71ae61bf9c0d\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " Oct 30 01:23:23.437323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.436837 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-podinfo\") pod \"2d82372c-ad61-4079-9412-71ae61bf9c0d\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " Oct 30 01:23:23.437323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.436868 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-envoy\") pod \"2d82372c-ad61-4079-9412-71ae61bf9c0d\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " Oct 30 01:23:23.437323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.436941 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-data\") pod \"2d82372c-ad61-4079-9412-71ae61bf9c0d\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " Oct 30 01:23:23.437323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.436987 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-credential-socket\") pod \"2d82372c-ad61-4079-9412-71ae61bf9c0d\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " Oct 30 01:23:23.437323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.437035 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-socket\") pod \"2d82372c-ad61-4079-9412-71ae61bf9c0d\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " Oct 30 01:23:23.437323 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.437094 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-token\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-token\") pod \"2d82372c-ad61-4079-9412-71ae61bf9c0d\" (UID: \"2d82372c-ad61-4079-9412-71ae61bf9c0d\") " Oct 30 01:23:23.440460 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.439610 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d82372c-ad61-4079-9412-71ae61bf9c0d-istiod-ca-cert" pod "2d82372c-ad61-4079-9412-71ae61bf9c0d" (UID: "2d82372c-ad61-4079-9412-71ae61bf9c0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:23.441183 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.441123 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-data" pod "2d82372c-ad61-4079-9412-71ae61bf9c0d" (UID: "2d82372c-ad61-4079-9412-71ae61bf9c0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:23.441498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.441357 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-token" pod "2d82372c-ad61-4079-9412-71ae61bf9c0d" (UID: "2d82372c-ad61-4079-9412-71ae61bf9c0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:23.441755 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.441703 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-credential-socket" pod "2d82372c-ad61-4079-9412-71ae61bf9c0d" (UID: "2d82372c-ad61-4079-9412-71ae61bf9c0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:23.441865 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.441790 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-socket" pod "2d82372c-ad61-4079-9412-71ae61bf9c0d" (UID: "2d82372c-ad61-4079-9412-71ae61bf9c0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:23.442178 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.442153 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-certs" pod "2d82372c-ad61-4079-9412-71ae61bf9c0d" (UID: "2d82372c-ad61-4079-9412-71ae61bf9c0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:23.442680 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.442625 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-podinfo" pod "2d82372c-ad61-4079-9412-71ae61bf9c0d" (UID: "2d82372c-ad61-4079-9412-71ae61bf9c0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:23.443936 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.443912 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-envoy" pod "2d82372c-ad61-4079-9412-71ae61bf9c0d" (UID: "2d82372c-ad61-4079-9412-71ae61bf9c0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:23.444219 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.444182 4927 generic.go:356] "Generic (PLEG): container finished" podID="ac0599b6-0434-4304-be95-553cd7dd0ba3" containerID="49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f" exitCode=0 Oct 30 01:23:23.444332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.444238 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-kube-api-access-9njft" pod "2d82372c-ad61-4079-9412-71ae61bf9c0d" (UID: "2d82372c-ad61-4079-9412-71ae61bf9c0d"). InnerVolumeSpecName "kube-api-access-9njft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:23.444404 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.444361 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" Oct 30 01:23:23.444505 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.444480 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" event={"ID":"ac0599b6-0434-4304-be95-553cd7dd0ba3","Type":"ContainerDied","Data":"49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f"} Oct 30 01:23:23.444660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.444590 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x" event={"ID":"ac0599b6-0434-4304-be95-553cd7dd0ba3","Type":"ContainerDied","Data":"768106c9116fb8e8622aebdadd1e0017d5c9ae1b0956f682e631e60233a4a10a"} Oct 30 01:23:23.444660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.444627 4927 scope.go:122] "RemoveContainer" containerID="49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f" Oct 30 01:23:23.448308 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.448187 4927 generic.go:356] "Generic (PLEG): container finished" podID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerID="618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2" exitCode=0 Oct 30 01:23:23.448308 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.448225 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" event={"ID":"2d82372c-ad61-4079-9412-71ae61bf9c0d","Type":"ContainerDied","Data":"618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2"} Oct 30 01:23:23.448308 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.448268 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" event={"ID":"2d82372c-ad61-4079-9412-71ae61bf9c0d","Type":"ContainerDied","Data":"b7c946636476715933d082ca9bf54335de9993cace43aad1510748d6611c7068"} Oct 30 01:23:23.448308 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.448287 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w" Oct 30 01:23:23.478554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.478534 4927 scope.go:122] "RemoveContainer" containerID="49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f" Oct 30 01:23:23.478870 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:23.478848 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f\": container with ID starting with 49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f not found: ID does not exist" containerID="49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f" Oct 30 01:23:23.478953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.478876 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f"} err="failed to get container status \"49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f\": rpc error: code = NotFound desc = could not find container \"49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f\": container with ID starting with 49c18f72487024cf4486fe40c99a5e9c752381145d3a7361d34b516f35f81c4f not found: ID does not exist" Oct 30 01:23:23.478953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.478895 4927 scope.go:122] "RemoveContainer" containerID="618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2" Oct 30 01:23:23.489133 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.489110 4927 scope.go:122] "RemoveContainer" containerID="618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2" Oct 30 01:23:23.489478 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:23.489456 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2\": container with ID starting with 618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2 not found: ID does not exist" containerID="618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2" Oct 30 01:23:23.489547 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.489487 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2"} err="failed to get container status \"618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2\": rpc error: code = NotFound desc = could not find container \"618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2\": container with ID starting with 618b39dc894633222681b93a58053a5385c6e094644cd88bff28be003d2112b2 not found: ID does not exist" Oct 30 01:23:23.513044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.513008 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x"] Oct 30 01:23:23.532433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.532400 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-889685dbc-q7v6x"] Oct 30 01:23:23.538932 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.538907 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.538932 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.538932 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d82372c-ad61-4079-9412-71ae61bf9c0d-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.539084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.538941 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.539084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.538949 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.539084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.538957 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.539084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.538986 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.539084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.538997 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d82372c-ad61-4079-9412-71ae61bf9c0d-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.539084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.539006 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.539084 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.539017 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njft\" (UniqueName: \"kubernetes.io/projected/2d82372c-ad61-4079-9412-71ae61bf9c0d-kube-api-access-9njft\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:23.574139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.574106 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5725f313-f8ee-40cd-9ef8-f17e18d8ab60" path="/var/lib/kubelet/pods/5725f313-f8ee-40cd-9ef8-f17e18d8ab60/volumes" Oct 30 01:23:23.574552 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.574538 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0599b6-0434-4304-be95-553cd7dd0ba3" path="/var/lib/kubelet/pods/ac0599b6-0434-4304-be95-553cd7dd0ba3/volumes" Oct 30 01:23:23.574968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.574953 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22f4c73-42a9-41b9-8f86-2dd9b4d41779" path="/var/lib/kubelet/pods/e22f4c73-42a9-41b9-8f86-2dd9b4d41779/volumes" Oct 30 01:23:23.575322 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.575305 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w"] Oct 30 01:23:23.575396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.575338 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-ckw5w"] Oct 30 01:23:23.849206 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.849118 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm"] Oct 30 01:23:23.849400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:23.849368 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" podUID="61512ccc-12bf-4206-9163-d11212cdba6d" containerName="maas-api" containerID="cri-o://477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b" gracePeriod=30 Oct 30 01:23:24.181395 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.181369 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 01:23:24.352964 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.351909 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls\") pod \"61512ccc-12bf-4206-9163-d11212cdba6d\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " Oct 30 01:23:24.352964 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.351985 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61512ccc-12bf-4206-9163-d11212cdba6d-kube-api-access-gb67v\" (UniqueName: \"kubernetes.io/projected/61512ccc-12bf-4206-9163-d11212cdba6d-kube-api-access-gb67v\") pod \"61512ccc-12bf-4206-9163-d11212cdba6d\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " Oct 30 01:23:24.352964 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.352025 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-metrics-certs\") pod \"61512ccc-12bf-4206-9163-d11212cdba6d\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " Oct 30 01:23:24.352964 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.352150 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61512ccc-12bf-4206-9163-d11212cdba6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61512ccc-12bf-4206-9163-d11212cdba6d-openshift-service-ca\") pod \"61512ccc-12bf-4206-9163-d11212cdba6d\" (UID: \"61512ccc-12bf-4206-9163-d11212cdba6d\") " Oct 30 01:23:24.353828 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.353471 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61512ccc-12bf-4206-9163-d11212cdba6d-openshift-service-ca" pod "61512ccc-12bf-4206-9163-d11212cdba6d" (UID: "61512ccc-12bf-4206-9163-d11212cdba6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:24.357411 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.357374 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61512ccc-12bf-4206-9163-d11212cdba6d-kube-api-access-gb67v" pod "61512ccc-12bf-4206-9163-d11212cdba6d" (UID: "61512ccc-12bf-4206-9163-d11212cdba6d"). InnerVolumeSpecName "kube-api-access-gb67v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:24.363831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.363804 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-metrics-certs" pod "61512ccc-12bf-4206-9163-d11212cdba6d" (UID: "61512ccc-12bf-4206-9163-d11212cdba6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:24.364946 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.364920 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls" pod "61512ccc-12bf-4206-9163-d11212cdba6d" (UID: "61512ccc-12bf-4206-9163-d11212cdba6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:24.454149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.454055 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61512ccc-12bf-4206-9163-d11212cdba6d-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:24.454149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.454090 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:24.454149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.454104 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb67v\" (UniqueName: \"kubernetes.io/projected/61512ccc-12bf-4206-9163-d11212cdba6d-kube-api-access-gb67v\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:24.454149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.454114 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61512ccc-12bf-4206-9163-d11212cdba6d-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:24.460165 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.460133 4927 generic.go:356] "Generic (PLEG): container finished" podID="61512ccc-12bf-4206-9163-d11212cdba6d" containerID="477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b" exitCode=0 Oct 30 01:23:24.460370 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.460186 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" Oct 30 01:23:24.460370 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.460233 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" event={"ID":"61512ccc-12bf-4206-9163-d11212cdba6d","Type":"ContainerDied","Data":"477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b"} Oct 30 01:23:24.460370 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.460299 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm" event={"ID":"61512ccc-12bf-4206-9163-d11212cdba6d","Type":"ContainerDied","Data":"c4ce26da8ee4c05661ff1eb98f54f3498b3d3452b5fce560780d5981db35019b"} Oct 30 01:23:24.460370 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.460335 4927 scope.go:122] "RemoveContainer" containerID="477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b" Oct 30 01:23:24.472820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.472794 4927 scope.go:122] "RemoveContainer" containerID="477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b" Oct 30 01:23:24.473176 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:24.473145 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b\": container with ID starting with 477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b not found: ID does not exist" containerID="477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b" Oct 30 01:23:24.473303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.473187 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b"} err="failed to get container status \"477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b\": rpc error: code = NotFound desc = could not find container \"477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b\": container with ID starting with 477582107b793f8ad20c82fb7f9ec6bea182e79a00da21854dd58e82cb1c900b not found: ID does not exist" Oct 30 01:23:24.510946 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.510903 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm"] Oct 30 01:23:24.521644 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:24.521617 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ea2547-74965dcd69-d5vrm"] Oct 30 01:23:25.573829 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:25.573799 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" path="/var/lib/kubelet/pods/2d82372c-ad61-4079-9412-71ae61bf9c0d/volumes" Oct 30 01:23:25.574268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:25.574252 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61512ccc-12bf-4206-9163-d11212cdba6d" path="/var/lib/kubelet/pods/61512ccc-12bf-4206-9163-d11212cdba6d/volumes" Oct 30 01:23:25.843402 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:25.843324 4927 scope.go:122] "RemoveContainer" containerID="163076a131f8a6395fc1e891f2b01cc5f9666547e61e0f2ab21eef5c5d2a36f2" Oct 30 01:23:31.912243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:31.912205 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8"] Oct 30 01:23:31.912771 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:31.912488 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" podUID="c0df2fee-b578-4ce6-995a-405e3edeeba3" containerName="maas-api" containerID="cri-o://2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f" gracePeriod=30 Oct 30 01:23:32.319652 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.319626 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:23:32.440611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.431057 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-maas-api-tls\") pod \"c0df2fee-b578-4ce6-995a-405e3edeeba3\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " Oct 30 01:23:32.440611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.431306 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0df2fee-b578-4ce6-995a-405e3edeeba3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df2fee-b578-4ce6-995a-405e3edeeba3-openshift-service-ca\") pod \"c0df2fee-b578-4ce6-995a-405e3edeeba3\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " Oct 30 01:23:32.440611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.431372 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-metrics-certs\") pod \"c0df2fee-b578-4ce6-995a-405e3edeeba3\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " Oct 30 01:23:32.440611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.431436 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0df2fee-b578-4ce6-995a-405e3edeeba3-kube-api-access-jngxv\" (UniqueName: \"kubernetes.io/projected/c0df2fee-b578-4ce6-995a-405e3edeeba3-kube-api-access-jngxv\") pod \"c0df2fee-b578-4ce6-995a-405e3edeeba3\" (UID: \"c0df2fee-b578-4ce6-995a-405e3edeeba3\") " Oct 30 01:23:32.440611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.432969 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0df2fee-b578-4ce6-995a-405e3edeeba3-openshift-service-ca" pod "c0df2fee-b578-4ce6-995a-405e3edeeba3" (UID: "c0df2fee-b578-4ce6-995a-405e3edeeba3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:32.440611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.436071 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-maas-api-tls" pod "c0df2fee-b578-4ce6-995a-405e3edeeba3" (UID: "c0df2fee-b578-4ce6-995a-405e3edeeba3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:32.462279 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.462233 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0df2fee-b578-4ce6-995a-405e3edeeba3-kube-api-access-jngxv" pod "c0df2fee-b578-4ce6-995a-405e3edeeba3" (UID: "c0df2fee-b578-4ce6-995a-405e3edeeba3"). InnerVolumeSpecName "kube-api-access-jngxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:32.489594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.484814 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-metrics-certs" pod "c0df2fee-b578-4ce6-995a-405e3edeeba3" (UID: "c0df2fee-b578-4ce6-995a-405e3edeeba3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:32.529530 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.529492 4927 generic.go:356] "Generic (PLEG): container finished" podID="c0df2fee-b578-4ce6-995a-405e3edeeba3" containerID="2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f" exitCode=0 Oct 30 01:23:32.529792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.529765 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" event={"ID":"c0df2fee-b578-4ce6-995a-405e3edeeba3","Type":"ContainerDied","Data":"2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f"} Oct 30 01:23:32.529870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.529852 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" event={"ID":"c0df2fee-b578-4ce6-995a-405e3edeeba3","Type":"ContainerDied","Data":"bc26b49dd35dcb8fd3bd77a11c737628bc6a10b86618f534aa12479f606adb70"} Oct 30 01:23:32.529934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.529771 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8" Oct 30 01:23:32.529934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.529873 4927 scope.go:122] "RemoveContainer" containerID="2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f" Oct 30 01:23:32.534456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.534427 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df2fee-b578-4ce6-995a-405e3edeeba3-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.534456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.534455 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.534691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.534468 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngxv\" (UniqueName: \"kubernetes.io/projected/c0df2fee-b578-4ce6-995a-405e3edeeba3-kube-api-access-jngxv\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.534691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.534482 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0df2fee-b578-4ce6-995a-405e3edeeba3-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.544420 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.544352 4927 scope.go:122] "RemoveContainer" containerID="2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f" Oct 30 01:23:32.544786 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:32.544765 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f\": container with ID starting with 2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f not found: ID does not exist" containerID="2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f" Oct 30 01:23:32.544873 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.544795 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f"} err="failed to get container status \"2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f\": rpc error: code = NotFound desc = could not find container \"2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f\": container with ID starting with 2ca05481052c0ee995f56d40d43bcaba60ca8a6ec572515aa80c041396733d8f not found: ID does not exist" Oct 30 01:23:32.580359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.580258 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8"] Oct 30 01:23:32.587052 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:32.587016 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae06fe-5847f6f5ff-8pgv8"] Oct 30 01:23:33.573139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:33.573103 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0df2fee-b578-4ce6-995a-405e3edeeba3" path="/var/lib/kubelet/pods/c0df2fee-b578-4ce6-995a-405e3edeeba3/volumes" Oct 30 01:23:34.443309 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:34.443265 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:23:40.071676 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:40.071616 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5"] Oct 30 01:23:40.072672 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:40.072611 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" containerID="cri-o://aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077" gracePeriod=30 Oct 30 01:23:42.316729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.316690 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx"] Oct 30 01:23:42.317581 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317551 4927 cpu_manager.go:414] "containerMap: removing container" podUID="61512ccc-12bf-4206-9163-d11212cdba6d" containerName="maas-api" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317603 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61512ccc-12bf-4206-9163-d11212cdba6d" containerName="maas-api" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317630 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ac0599b6-0434-4304-be95-553cd7dd0ba3" containerName="payload-pre-processing" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317639 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0599b6-0434-4304-be95-553cd7dd0ba3" containerName="payload-pre-processing" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317655 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317663 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317686 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e22f4c73-42a9-41b9-8f86-2dd9b4d41779" containerName="payload-pre-processing" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317694 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22f4c73-42a9-41b9-8f86-2dd9b4d41779" containerName="payload-pre-processing" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317716 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c0df2fee-b578-4ce6-995a-405e3edeeba3" containerName="maas-api" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317722 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0df2fee-b578-4ce6-995a-405e3edeeba3" containerName="maas-api" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317728 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerName="istio-proxy" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317733 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerName="istio-proxy" Oct 30 01:23:42.317747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317754 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5725f313-f8ee-40cd-9ef8-f17e18d8ab60" containerName="payload-processing" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317762 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5725f313-f8ee-40cd-9ef8-f17e18d8ab60" containerName="payload-processing" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317793 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f9676b8e-5fb3-438d-8138-b6f906d3dd52" containerName="maas-api" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317801 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9676b8e-5fb3-438d-8138-b6f906d3dd52" containerName="maas-api" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.317991 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0599b6-0434-4304-be95-553cd7dd0ba3" containerName="payload-pre-processing" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.318005 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4378ab-80c5-42d8-a817-8c11650d094b" containerName="istio-proxy" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.318019 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22f4c73-42a9-41b9-8f86-2dd9b4d41779" containerName="payload-pre-processing" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.318033 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0df2fee-b578-4ce6-995a-405e3edeeba3" containerName="maas-api" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.318040 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d82372c-ad61-4079-9412-71ae61bf9c0d" containerName="istio-proxy" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.318047 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5725f313-f8ee-40cd-9ef8-f17e18d8ab60" containerName="payload-processing" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.318056 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9676b8e-5fb3-438d-8138-b6f906d3dd52" containerName="maas-api" Oct 30 01:23:42.318302 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.318066 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="61512ccc-12bf-4206-9163-d11212cdba6d" containerName="maas-api" Oct 30 01:23:42.323233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.323214 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:42.346805 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.346768 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx"] Oct 30 01:23:42.348978 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.348951 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3e9921\"" Oct 30 01:23:42.349122 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.349106 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3e9921-dockercfg-48d8b\"" Oct 30 01:23:42.412955 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.412917 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414818dd-4954-4548-be30-01f1a9ded7c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx\" (UID: \"414818dd-4954-4548-be30-01f1a9ded7c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:42.413153 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.413081 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpjf\" (UniqueName: \"kubernetes.io/projected/414818dd-4954-4548-be30-01f1a9ded7c5-kube-api-access-crpjf\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx\" (UID: \"414818dd-4954-4548-be30-01f1a9ded7c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:42.514432 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.514392 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414818dd-4954-4548-be30-01f1a9ded7c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx\" (UID: \"414818dd-4954-4548-be30-01f1a9ded7c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:42.514651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.514527 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpjf\" (UniqueName: \"kubernetes.io/projected/414818dd-4954-4548-be30-01f1a9ded7c5-kube-api-access-crpjf\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx\" (UID: \"414818dd-4954-4548-be30-01f1a9ded7c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:42.515356 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.515326 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414818dd-4954-4548-be30-01f1a9ded7c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx\" (UID: \"414818dd-4954-4548-be30-01f1a9ded7c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:42.537774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.537744 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpjf\" (UniqueName: \"kubernetes.io/projected/414818dd-4954-4548-be30-01f1a9ded7c5-kube-api-access-crpjf\") pod \"payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx\" (UID: \"414818dd-4954-4548-be30-01f1a9ded7c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:42.658864 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.658812 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:42.966184 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:42.966149 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx"] Oct 30 01:23:42.967550 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:23:42.967516 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414818dd_4954_4548_be30_01f1a9ded7c5.slice/crio-512af93ff8ec821283671c06191f7f0ae5b5a5b9507dbccefcb573edc5f41949 WatchSource:0}: Error finding container 512af93ff8ec821283671c06191f7f0ae5b5a5b9507dbccefcb573edc5f41949: Status 404 returned error can't find the container with id 512af93ff8ec821283671c06191f7f0ae5b5a5b9507dbccefcb573edc5f41949 Oct 30 01:23:43.267417 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:43.267340 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8"] Oct 30 01:23:43.616136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:43.616056 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" event={"ID":"414818dd-4954-4548-be30-01f1a9ded7c5","Type":"ContainerStarted","Data":"3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d"} Oct 30 01:23:43.616136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:43.616111 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" event={"ID":"414818dd-4954-4548-be30-01f1a9ded7c5","Type":"ContainerStarted","Data":"512af93ff8ec821283671c06191f7f0ae5b5a5b9507dbccefcb573edc5f41949"} Oct 30 01:23:43.616550 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:43.616176 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:43.658533 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:43.658471 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" podStartSLOduration=1.658457731 podStartE2EDuration="1.658457731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:43.655061412 +0000 UTC m=+2962.226987635" watchObservedRunningTime="2026-10-30 01:23:43.658457731 +0000 UTC m=+2962.230383955" Oct 30 01:23:45.368048 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.368017 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:23:45.457619 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.457554 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-envoy\") pod \"1ffb6932-005f-4829-a8c0-9ff8801be789\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " Oct 30 01:23:45.457874 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.457703 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-token\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-token\") pod \"1ffb6932-005f-4829-a8c0-9ff8801be789\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " Oct 30 01:23:45.458170 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.458143 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-socket\") pod \"1ffb6932-005f-4829-a8c0-9ff8801be789\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " Oct 30 01:23:45.458307 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.458196 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-kube-api-access-lq8bf\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-kube-api-access-lq8bf\") pod \"1ffb6932-005f-4829-a8c0-9ff8801be789\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " Oct 30 01:23:45.458307 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.458254 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-certs\") pod \"1ffb6932-005f-4829-a8c0-9ff8801be789\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " Oct 30 01:23:45.458432 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.458365 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffb6932-005f-4829-a8c0-9ff8801be789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffb6932-005f-4829-a8c0-9ff8801be789-istiod-ca-cert\") pod \"1ffb6932-005f-4829-a8c0-9ff8801be789\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " Oct 30 01:23:45.458432 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.458414 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-credential-socket\") pod \"1ffb6932-005f-4829-a8c0-9ff8801be789\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " Oct 30 01:23:45.458556 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.458480 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-data\") pod \"1ffb6932-005f-4829-a8c0-9ff8801be789\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " Oct 30 01:23:45.458658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.458562 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-podinfo\") pod \"1ffb6932-005f-4829-a8c0-9ff8801be789\" (UID: \"1ffb6932-005f-4829-a8c0-9ff8801be789\") " Oct 30 01:23:45.460303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.459813 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-data" pod "1ffb6932-005f-4829-a8c0-9ff8801be789" (UID: "1ffb6932-005f-4829-a8c0-9ff8801be789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.460303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.460019 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-credential-socket" pod "1ffb6932-005f-4829-a8c0-9ff8801be789" (UID: "1ffb6932-005f-4829-a8c0-9ff8801be789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.460303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.460034 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-certs" pod "1ffb6932-005f-4829-a8c0-9ff8801be789" (UID: "1ffb6932-005f-4829-a8c0-9ff8801be789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.460303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.460084 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffb6932-005f-4829-a8c0-9ff8801be789-istiod-ca-cert" pod "1ffb6932-005f-4829-a8c0-9ff8801be789" (UID: "1ffb6932-005f-4829-a8c0-9ff8801be789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:45.460303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.460257 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-envoy" pod "1ffb6932-005f-4829-a8c0-9ff8801be789" (UID: "1ffb6932-005f-4829-a8c0-9ff8801be789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.460709 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.460313 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-socket" pod "1ffb6932-005f-4829-a8c0-9ff8801be789" (UID: "1ffb6932-005f-4829-a8c0-9ff8801be789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:45.462537 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.462229 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-token" pod "1ffb6932-005f-4829-a8c0-9ff8801be789" (UID: "1ffb6932-005f-4829-a8c0-9ff8801be789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:45.462537 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.462292 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-kube-api-access-lq8bf" pod "1ffb6932-005f-4829-a8c0-9ff8801be789" (UID: "1ffb6932-005f-4829-a8c0-9ff8801be789"). InnerVolumeSpecName "kube-api-access-lq8bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:45.462537 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.462329 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-podinfo" pod "1ffb6932-005f-4829-a8c0-9ff8801be789" (UID: "1ffb6932-005f-4829-a8c0-9ff8801be789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:45.560897 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.560859 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.560897 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.560892 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.560897 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.560902 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq8bf\" (UniqueName: \"kubernetes.io/projected/1ffb6932-005f-4829-a8c0-9ff8801be789-kube-api-access-lq8bf\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.560897 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.560912 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.561289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.560921 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffb6932-005f-4829-a8c0-9ff8801be789-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.561289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.560929 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.561289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.560939 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.561289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.560951 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.561289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.560964 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffb6932-005f-4829-a8c0-9ff8801be789-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:45.631753 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.631713 4927 generic.go:356] "Generic (PLEG): container finished" podID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerID="aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077" exitCode=0 Oct 30 01:23:45.631948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.631767 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" Oct 30 01:23:45.631948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.631797 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" event={"ID":"1ffb6932-005f-4829-a8c0-9ff8801be789","Type":"ContainerDied","Data":"aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077"} Oct 30 01:23:45.631948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.631852 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5" event={"ID":"1ffb6932-005f-4829-a8c0-9ff8801be789","Type":"ContainerDied","Data":"e8cc0a20ba7287312a7acbd56a35739032da43ef8d983a3eb22e502cf2e11400"} Oct 30 01:23:45.631948 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.631874 4927 scope.go:122] "RemoveContainer" containerID="aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077" Oct 30 01:23:45.643172 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.643156 4927 scope.go:122] "RemoveContainer" containerID="aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077" Oct 30 01:23:45.643439 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:45.643408 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077\": container with ID starting with aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077 not found: ID does not exist" containerID="aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077" Oct 30 01:23:45.643487 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.643446 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077"} err="failed to get container status \"aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077\": rpc error: code = NotFound desc = could not find container \"aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077\": container with ID starting with aa2f3153dfcdef84150e58da587b56bc11845f422f53308d49b5b6852631f077 not found: ID does not exist" Oct 30 01:23:45.686033 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.686009 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5"] Oct 30 01:23:45.692444 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:45.692415 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-dx9j5"] Oct 30 01:23:47.572259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:47.572220 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" path="/var/lib/kubelet/pods/1ffb6932-005f-4829-a8c0-9ff8801be789/volumes" Oct 30 01:23:48.268683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:48.268610 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" podUID="97acea83-49aa-4c90-bf1c-312dc1dd7d02" containerName="payload-processing" containerID="cri-o://06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc" gracePeriod=25 Oct 30 01:23:49.366841 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:49.366806 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999"] Oct 30 01:23:49.367391 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:49.367009 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" podUID="b04c018e-d268-4c73-97c2-63436cebfd55" containerName="maas-api" containerID="cri-o://ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df" gracePeriod=30 Oct 30 01:23:49.948035 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:49.948000 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:23:50.029689 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.029651 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-metrics-certs\") pod \"b04c018e-d268-4c73-97c2-63436cebfd55\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " Oct 30 01:23:50.029876 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.029806 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-maas-api-tls\") pod \"b04c018e-d268-4c73-97c2-63436cebfd55\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " Oct 30 01:23:50.029955 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.029878 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04c018e-d268-4c73-97c2-63436cebfd55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04c018e-d268-4c73-97c2-63436cebfd55-openshift-service-ca\") pod \"b04c018e-d268-4c73-97c2-63436cebfd55\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " Oct 30 01:23:50.030054 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.030035 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04c018e-d268-4c73-97c2-63436cebfd55-kube-api-access-wvzf2\" (UniqueName: \"kubernetes.io/projected/b04c018e-d268-4c73-97c2-63436cebfd55-kube-api-access-wvzf2\") pod \"b04c018e-d268-4c73-97c2-63436cebfd55\" (UID: \"b04c018e-d268-4c73-97c2-63436cebfd55\") " Oct 30 01:23:50.030381 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.030285 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04c018e-d268-4c73-97c2-63436cebfd55-openshift-service-ca" pod "b04c018e-d268-4c73-97c2-63436cebfd55" (UID: "b04c018e-d268-4c73-97c2-63436cebfd55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:50.030646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.030554 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04c018e-d268-4c73-97c2-63436cebfd55-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:50.032259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.032229 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-maas-api-tls" pod "b04c018e-d268-4c73-97c2-63436cebfd55" (UID: "b04c018e-d268-4c73-97c2-63436cebfd55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:50.032471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.032450 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-metrics-certs" pod "b04c018e-d268-4c73-97c2-63436cebfd55" (UID: "b04c018e-d268-4c73-97c2-63436cebfd55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:50.032628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.032560 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04c018e-d268-4c73-97c2-63436cebfd55-kube-api-access-wvzf2" pod "b04c018e-d268-4c73-97c2-63436cebfd55" (UID: "b04c018e-d268-4c73-97c2-63436cebfd55"). InnerVolumeSpecName "kube-api-access-wvzf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:50.132287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.132251 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:50.132287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.132282 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzf2\" (UniqueName: \"kubernetes.io/projected/b04c018e-d268-4c73-97c2-63436cebfd55-kube-api-access-wvzf2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:50.132287 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.132292 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04c018e-d268-4c73-97c2-63436cebfd55-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:50.671490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.671454 4927 generic.go:356] "Generic (PLEG): container finished" podID="b04c018e-d268-4c73-97c2-63436cebfd55" containerID="ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df" exitCode=0 Oct 30 01:23:50.671996 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.671503 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" Oct 30 01:23:50.671996 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.671540 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" event={"ID":"b04c018e-d268-4c73-97c2-63436cebfd55","Type":"ContainerDied","Data":"ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df"} Oct 30 01:23:50.671996 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.671622 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999" event={"ID":"b04c018e-d268-4c73-97c2-63436cebfd55","Type":"ContainerDied","Data":"7ec17d7bec41101b81c51ed77685b0dee7584b1afc995d20844146af4cf087bb"} Oct 30 01:23:50.671996 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.671647 4927 scope.go:122] "RemoveContainer" containerID="ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df" Oct 30 01:23:50.683842 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.683789 4927 scope.go:122] "RemoveContainer" containerID="ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df" Oct 30 01:23:50.684081 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:50.684060 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df\": container with ID starting with ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df not found: ID does not exist" containerID="ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df" Oct 30 01:23:50.684160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.684088 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df"} err="failed to get container status \"ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df\": rpc error: code = NotFound desc = could not find container \"ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df\": container with ID starting with ac492248b1e3df2e44c5c05fbfa3a18c3fccab1b96a736efd00e90208d9488df not found: ID does not exist" Oct 30 01:23:50.722749 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.722719 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999"] Oct 30 01:23:50.734010 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:50.733984 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abf8ae-7d6f6556f4-xm999"] Oct 30 01:23:51.186080 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:51.186046 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq"] Oct 30 01:23:51.219223 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:51.219187 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c"] Oct 30 01:23:51.572594 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:51.572500 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04c018e-d268-4c73-97c2-63436cebfd55" path="/var/lib/kubelet/pods/b04c018e-d268-4c73-97c2-63436cebfd55/volumes" Oct 30 01:23:54.625745 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:54.625710 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:23:54.808645 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:54.808596 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" podUID="97acea83-49aa-4c90-bf1c-312dc1dd7d02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:55.742012 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:55.741971 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc"] Oct 30 01:23:56.187365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:56.187308 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" podUID="65ce4978-b441-48c9-9145-24c6bdf788f6" containerName="payload-processing" containerID="cri-o://ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7" gracePeriod=25 Oct 30 01:23:56.220283 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:56.220226 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" podUID="cd70c464-1507-414d-96d9-a138e4143281" containerName="payload-pre-processing" containerID="cri-o://bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985" gracePeriod=25 Oct 30 01:23:57.754776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:57.754737 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm"] Oct 30 01:23:57.755272 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:57.754981 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" podUID="5a52d418-6bef-48df-841e-5b55dd4e42ac" containerName="maas-api" containerID="cri-o://0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037" gracePeriod=30 Oct 30 01:23:58.071293 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.071257 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 01:23:58.153597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.151802 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a52d418-6bef-48df-841e-5b55dd4e42ac-kube-api-access-wqfw4\" (UniqueName: \"kubernetes.io/projected/5a52d418-6bef-48df-841e-5b55dd4e42ac-kube-api-access-wqfw4\") pod \"5a52d418-6bef-48df-841e-5b55dd4e42ac\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " Oct 30 01:23:58.153597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.151964 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-maas-api-tls\") pod \"5a52d418-6bef-48df-841e-5b55dd4e42ac\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " Oct 30 01:23:58.153597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.152025 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a52d418-6bef-48df-841e-5b55dd4e42ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a52d418-6bef-48df-841e-5b55dd4e42ac-openshift-service-ca\") pod \"5a52d418-6bef-48df-841e-5b55dd4e42ac\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " Oct 30 01:23:58.153597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.152166 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-metrics-certs\") pod \"5a52d418-6bef-48df-841e-5b55dd4e42ac\" (UID: \"5a52d418-6bef-48df-841e-5b55dd4e42ac\") " Oct 30 01:23:58.186247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.185884 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-maas-api-tls" pod "5a52d418-6bef-48df-841e-5b55dd4e42ac" (UID: "5a52d418-6bef-48df-841e-5b55dd4e42ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:58.191532 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.191494 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a52d418-6bef-48df-841e-5b55dd4e42ac-openshift-service-ca" pod "5a52d418-6bef-48df-841e-5b55dd4e42ac" (UID: "5a52d418-6bef-48df-841e-5b55dd4e42ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:58.193056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.193021 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-metrics-certs" pod "5a52d418-6bef-48df-841e-5b55dd4e42ac" (UID: "5a52d418-6bef-48df-841e-5b55dd4e42ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:58.200127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.200097 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a52d418-6bef-48df-841e-5b55dd4e42ac-kube-api-access-wqfw4" pod "5a52d418-6bef-48df-841e-5b55dd4e42ac" (UID: "5a52d418-6bef-48df-841e-5b55dd4e42ac"). InnerVolumeSpecName "kube-api-access-wqfw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:58.254178 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.254130 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:58.254178 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.254164 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a52d418-6bef-48df-841e-5b55dd4e42ac-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:58.254178 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.254177 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a52d418-6bef-48df-841e-5b55dd4e42ac-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:58.254178 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.254186 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqfw4\" (UniqueName: \"kubernetes.io/projected/5a52d418-6bef-48df-841e-5b55dd4e42ac-kube-api-access-wqfw4\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:23:58.734238 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.734205 4927 generic.go:356] "Generic (PLEG): container finished" podID="5a52d418-6bef-48df-841e-5b55dd4e42ac" containerID="0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037" exitCode=0 Oct 30 01:23:58.734429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.734251 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" Oct 30 01:23:58.734429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.734288 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" event={"ID":"5a52d418-6bef-48df-841e-5b55dd4e42ac","Type":"ContainerDied","Data":"0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037"} Oct 30 01:23:58.734429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.734348 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm" event={"ID":"5a52d418-6bef-48df-841e-5b55dd4e42ac","Type":"ContainerDied","Data":"d4b70fd7a9e02d4b03fa240d277bd405f1bd297294d78e88bccb627b115ea94b"} Oct 30 01:23:58.734429 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.734371 4927 scope.go:122] "RemoveContainer" containerID="0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037" Oct 30 01:23:58.746379 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.746357 4927 scope.go:122] "RemoveContainer" containerID="0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037" Oct 30 01:23:58.746819 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:23:58.746783 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037\": container with ID starting with 0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037 not found: ID does not exist" containerID="0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037" Oct 30 01:23:58.746952 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.746829 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037"} err="failed to get container status \"0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037\": rpc error: code = NotFound desc = could not find container \"0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037\": container with ID starting with 0a272e8d51cc52f7239c16b439862013fc78bdde6aa9f0eefcdb8535e9612037 not found: ID does not exist" Oct 30 01:23:58.793479 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.793445 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm"] Oct 30 01:23:58.803155 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:58.803124 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-60e8f4-74f574bfc5-tfmfm"] Oct 30 01:23:59.572017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:23:59.571985 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a52d418-6bef-48df-841e-5b55dd4e42ac" path="/var/lib/kubelet/pods/5a52d418-6bef-48df-841e-5b55dd4e42ac/volumes" Oct 30 01:24:01.291232 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:01.291195 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" podUID="65ce4978-b441-48c9-9145-24c6bdf788f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:01.291232 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:01.291214 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" podUID="cd70c464-1507-414d-96d9-a138e4143281" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:04.442010 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:04.441968 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:24:04.807848 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:04.807756 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" podUID="97acea83-49aa-4c90-bf1c-312dc1dd7d02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:08.543713 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.543652 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 01:24:08.699501 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.699464 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97acea83-49aa-4c90-bf1c-312dc1dd7d02-kube-api-access-xkjvg\" (UniqueName: \"kubernetes.io/projected/97acea83-49aa-4c90-bf1c-312dc1dd7d02-kube-api-access-xkjvg\") pod \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\" (UID: \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\") " Oct 30 01:24:08.699748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.699530 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97acea83-49aa-4c90-bf1c-312dc1dd7d02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97acea83-49aa-4c90-bf1c-312dc1dd7d02-plugins-config-volume\") pod \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\" (UID: \"97acea83-49aa-4c90-bf1c-312dc1dd7d02\") " Oct 30 01:24:08.699937 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.699911 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97acea83-49aa-4c90-bf1c-312dc1dd7d02-plugins-config-volume" pod "97acea83-49aa-4c90-bf1c-312dc1dd7d02" (UID: "97acea83-49aa-4c90-bf1c-312dc1dd7d02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:08.701781 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.701759 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97acea83-49aa-4c90-bf1c-312dc1dd7d02-kube-api-access-xkjvg" pod "97acea83-49aa-4c90-bf1c-312dc1dd7d02" (UID: "97acea83-49aa-4c90-bf1c-312dc1dd7d02"). InnerVolumeSpecName "kube-api-access-xkjvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:08.801812 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.801727 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkjvg\" (UniqueName: \"kubernetes.io/projected/97acea83-49aa-4c90-bf1c-312dc1dd7d02-kube-api-access-xkjvg\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:08.801812 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.801757 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97acea83-49aa-4c90-bf1c-312dc1dd7d02-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:08.814655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.814623 4927 generic.go:356] "Generic (PLEG): container finished" podID="97acea83-49aa-4c90-bf1c-312dc1dd7d02" containerID="06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc" exitCode=0 Oct 30 01:24:08.814831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.814675 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" Oct 30 01:24:08.814831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.814710 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" event={"ID":"97acea83-49aa-4c90-bf1c-312dc1dd7d02","Type":"ContainerDied","Data":"06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc"} Oct 30 01:24:08.814831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.814774 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8" event={"ID":"97acea83-49aa-4c90-bf1c-312dc1dd7d02","Type":"ContainerDied","Data":"59dbf460e1e42e7281b999d778b9007e78a5c43135145b45dec984467746eda0"} Oct 30 01:24:08.814831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.814804 4927 scope.go:122] "RemoveContainer" containerID="06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc" Oct 30 01:24:08.825900 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.825880 4927 scope.go:122] "RemoveContainer" containerID="06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc" Oct 30 01:24:08.826219 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:08.826194 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc\": container with ID starting with 06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc not found: ID does not exist" containerID="06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc" Oct 30 01:24:08.826374 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.826229 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc"} err="failed to get container status \"06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc\": rpc error: code = NotFound desc = could not find container \"06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc\": container with ID starting with 06370b75252c6dbdb0030ef2a2a08b41b4a9f7fa08a06e8e0a64ae87a9605bcc not found: ID does not exist" Oct 30 01:24:08.881477 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.881417 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8"] Oct 30 01:24:08.885316 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:08.885278 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-60e8f4-69fc4846cb-7whf8"] Oct 30 01:24:09.572112 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:09.572076 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97acea83-49aa-4c90-bf1c-312dc1dd7d02" path="/var/lib/kubelet/pods/97acea83-49aa-4c90-bf1c-312dc1dd7d02/volumes" Oct 30 01:24:10.767451 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:10.767405 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" podUID="888a0308-4626-478a-80c7-efcdcb04cf19" containerName="main" containerID="cri-o://be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121" gracePeriod=45 Oct 30 01:24:11.049153 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.049126 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:24:11.234962 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.234933 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888a0308-4626-478a-80c7-efcdcb04cf19-kube-api-access-vrbrv\" (UniqueName: \"kubernetes.io/projected/888a0308-4626-478a-80c7-efcdcb04cf19-kube-api-access-vrbrv\") pod \"888a0308-4626-478a-80c7-efcdcb04cf19\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " Oct 30 01:24:11.235150 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235033 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-model-cache\") pod \"888a0308-4626-478a-80c7-efcdcb04cf19\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " Oct 30 01:24:11.235150 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235123 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/888a0308-4626-478a-80c7-efcdcb04cf19-tls-certs\" (UniqueName: \"kubernetes.io/secret/888a0308-4626-478a-80c7-efcdcb04cf19-tls-certs\") pod \"888a0308-4626-478a-80c7-efcdcb04cf19\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " Oct 30 01:24:11.235229 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235181 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-home\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-home\") pod \"888a0308-4626-478a-80c7-efcdcb04cf19\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " Oct 30 01:24:11.235265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235239 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-dshm\") pod \"888a0308-4626-478a-80c7-efcdcb04cf19\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " Oct 30 01:24:11.235319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235268 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-tmp-dir\") pod \"888a0308-4626-478a-80c7-efcdcb04cf19\" (UID: \"888a0308-4626-478a-80c7-efcdcb04cf19\") " Oct 30 01:24:11.235391 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235355 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-model-cache" pod "888a0308-4626-478a-80c7-efcdcb04cf19" (UID: "888a0308-4626-478a-80c7-efcdcb04cf19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.235562 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235507 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-home" pod "888a0308-4626-478a-80c7-efcdcb04cf19" (UID: "888a0308-4626-478a-80c7-efcdcb04cf19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.235700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235608 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-tmp-dir" pod "888a0308-4626-478a-80c7-efcdcb04cf19" (UID: "888a0308-4626-478a-80c7-efcdcb04cf19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.235850 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235830 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.235933 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235853 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.235933 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.235863 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.237375 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.237352 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-dshm" pod "888a0308-4626-478a-80c7-efcdcb04cf19" (UID: "888a0308-4626-478a-80c7-efcdcb04cf19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.237474 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.237399 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888a0308-4626-478a-80c7-efcdcb04cf19-kube-api-access-vrbrv" pod "888a0308-4626-478a-80c7-efcdcb04cf19" (UID: "888a0308-4626-478a-80c7-efcdcb04cf19"). InnerVolumeSpecName "kube-api-access-vrbrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:11.237474 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.237447 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/888a0308-4626-478a-80c7-efcdcb04cf19-tls-certs" pod "888a0308-4626-478a-80c7-efcdcb04cf19" (UID: "888a0308-4626-478a-80c7-efcdcb04cf19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:11.291300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.291227 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" podUID="cd70c464-1507-414d-96d9-a138e4143281" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:11.291300 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.291236 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" podUID="65ce4978-b441-48c9-9145-24c6bdf788f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:11.337322 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.337292 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888a0308-4626-478a-80c7-efcdcb04cf19-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.337322 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.337318 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888a0308-4626-478a-80c7-efcdcb04cf19-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.337322 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.337327 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrbrv\" (UniqueName: \"kubernetes.io/projected/888a0308-4626-478a-80c7-efcdcb04cf19-kube-api-access-vrbrv\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.840727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.840635 4927 generic.go:356] "Generic (PLEG): container finished" podID="888a0308-4626-478a-80c7-efcdcb04cf19" containerID="be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121" exitCode=0 Oct 30 01:24:11.840727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.840674 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" Oct 30 01:24:11.840727 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.840704 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" event={"ID":"888a0308-4626-478a-80c7-efcdcb04cf19","Type":"ContainerDied","Data":"be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121"} Oct 30 01:24:11.841276 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.840751 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc" event={"ID":"888a0308-4626-478a-80c7-efcdcb04cf19","Type":"ContainerDied","Data":"b84fbde7b43554d38b3bc17d18a15ba9584954b945a6a9fcaeda0be895612389"} Oct 30 01:24:11.841276 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.840774 4927 scope.go:122] "RemoveContainer" containerID="be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121" Oct 30 01:24:11.851180 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.851165 4927 scope.go:122] "RemoveContainer" containerID="be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121" Oct 30 01:24:11.851470 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:11.851447 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121\": container with ID starting with be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121 not found: ID does not exist" containerID="be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121" Oct 30 01:24:11.851540 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.851477 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121"} err="failed to get container status \"be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121\": rpc error: code = NotFound desc = could not find container \"be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121\": container with ID starting with be77926c87d656e6ea595154c3690a22a481286f056c1cec1b34a424e4499121 not found: ID does not exist" Oct 30 01:24:11.889953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.889923 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc"] Oct 30 01:24:11.900984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:11.900957 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-abf8ae/ipp-route-abf8ae-kserve-55d6f9c697-h4hlc"] Oct 30 01:24:13.571620 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:13.571561 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888a0308-4626-478a-80c7-efcdcb04cf19" path="/var/lib/kubelet/pods/888a0308-4626-478a-80c7-efcdcb04cf19/volumes" Oct 30 01:24:15.048367 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.048328 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4"] Oct 30 01:24:15.049121 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.049084 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" podUID="29ab4ce5-416f-4f47-832a-da72d1263042" containerName="maas-api" containerID="cri-o://06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865" gracePeriod=30 Oct 30 01:24:15.202957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.202918 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8"] Oct 30 01:24:15.203203 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.203177 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" podUID="82cb426a-d010-427a-a153-569fee89d6a7" containerName="payload-processing" containerID="cri-o://6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673" gracePeriod=30 Oct 30 01:24:15.222990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.222957 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv"] Oct 30 01:24:15.223400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.223368 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" podUID="1cd85074-3f4b-4d36-ad24-fa7abc24a367" containerName="payload-pre-processing" containerID="cri-o://c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04" gracePeriod=30 Oct 30 01:24:15.364901 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.364870 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:24:15.385424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.385386 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-maas-api-tls\") pod \"29ab4ce5-416f-4f47-832a-da72d1263042\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " Oct 30 01:24:15.385617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.385465 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-metrics-certs\") pod \"29ab4ce5-416f-4f47-832a-da72d1263042\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " Oct 30 01:24:15.385617 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.385560 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ab4ce5-416f-4f47-832a-da72d1263042-kube-api-access-7vhjw\" (UniqueName: \"kubernetes.io/projected/29ab4ce5-416f-4f47-832a-da72d1263042-kube-api-access-7vhjw\") pod \"29ab4ce5-416f-4f47-832a-da72d1263042\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " Oct 30 01:24:15.385722 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.385671 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ab4ce5-416f-4f47-832a-da72d1263042-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab4ce5-416f-4f47-832a-da72d1263042-openshift-service-ca\") pod \"29ab4ce5-416f-4f47-832a-da72d1263042\" (UID: \"29ab4ce5-416f-4f47-832a-da72d1263042\") " Oct 30 01:24:15.386565 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.386515 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ab4ce5-416f-4f47-832a-da72d1263042-openshift-service-ca" pod "29ab4ce5-416f-4f47-832a-da72d1263042" (UID: "29ab4ce5-416f-4f47-832a-da72d1263042"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:15.388540 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.388512 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-metrics-certs" pod "29ab4ce5-416f-4f47-832a-da72d1263042" (UID: "29ab4ce5-416f-4f47-832a-da72d1263042"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:15.388974 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.388945 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-maas-api-tls" pod "29ab4ce5-416f-4f47-832a-da72d1263042" (UID: "29ab4ce5-416f-4f47-832a-da72d1263042"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:15.389260 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.389237 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ab4ce5-416f-4f47-832a-da72d1263042-kube-api-access-7vhjw" pod "29ab4ce5-416f-4f47-832a-da72d1263042" (UID: "29ab4ce5-416f-4f47-832a-da72d1263042"). InnerVolumeSpecName "kube-api-access-7vhjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:15.487946 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.487908 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vhjw\" (UniqueName: \"kubernetes.io/projected/29ab4ce5-416f-4f47-832a-da72d1263042-kube-api-access-7vhjw\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:15.488110 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.487951 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab4ce5-416f-4f47-832a-da72d1263042-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:15.488110 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.487968 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:15.488110 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.487982 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab4ce5-416f-4f47-832a-da72d1263042-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:15.502551 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.502204 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:24:15.636240 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.636202 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:24:15.718722 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.718672 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cb426a-d010-427a-a153-569fee89d6a7-plugins-config-volume" pod "82cb426a-d010-427a-a153-569fee89d6a7" (UID: "82cb426a-d010-427a-a153-569fee89d6a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:15.719167 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.718444 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82cb426a-d010-427a-a153-569fee89d6a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb426a-d010-427a-a153-569fee89d6a7-plugins-config-volume\") pod \"82cb426a-d010-427a-a153-569fee89d6a7\" (UID: \"82cb426a-d010-427a-a153-569fee89d6a7\") " Oct 30 01:24:15.719449 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.719427 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82cb426a-d010-427a-a153-569fee89d6a7-kube-api-access-566rd\" (UniqueName: \"kubernetes.io/projected/82cb426a-d010-427a-a153-569fee89d6a7-kube-api-access-566rd\") pod \"82cb426a-d010-427a-a153-569fee89d6a7\" (UID: \"82cb426a-d010-427a-a153-569fee89d6a7\") " Oct 30 01:24:15.719973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.719946 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cb426a-d010-427a-a153-569fee89d6a7-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:15.722371 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.722346 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cb426a-d010-427a-a153-569fee89d6a7-kube-api-access-566rd" pod "82cb426a-d010-427a-a153-569fee89d6a7" (UID: "82cb426a-d010-427a-a153-569fee89d6a7"). InnerVolumeSpecName "kube-api-access-566rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:15.820920 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.820881 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd85074-3f4b-4d36-ad24-fa7abc24a367-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd85074-3f4b-4d36-ad24-fa7abc24a367-plugins-config-volume\") pod \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\" (UID: \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\") " Oct 30 01:24:15.821134 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.821071 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd85074-3f4b-4d36-ad24-fa7abc24a367-kube-api-access-dbf8h\" (UniqueName: \"kubernetes.io/projected/1cd85074-3f4b-4d36-ad24-fa7abc24a367-kube-api-access-dbf8h\") pod \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\" (UID: \"1cd85074-3f4b-4d36-ad24-fa7abc24a367\") " Oct 30 01:24:15.821276 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.821251 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd85074-3f4b-4d36-ad24-fa7abc24a367-plugins-config-volume" pod "1cd85074-3f4b-4d36-ad24-fa7abc24a367" (UID: "1cd85074-3f4b-4d36-ad24-fa7abc24a367"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:15.821547 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.821528 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-566rd\" (UniqueName: \"kubernetes.io/projected/82cb426a-d010-427a-a153-569fee89d6a7-kube-api-access-566rd\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:15.821658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.821550 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd85074-3f4b-4d36-ad24-fa7abc24a367-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:15.823248 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.823222 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd85074-3f4b-4d36-ad24-fa7abc24a367-kube-api-access-dbf8h" pod "1cd85074-3f4b-4d36-ad24-fa7abc24a367" (UID: "1cd85074-3f4b-4d36-ad24-fa7abc24a367"). InnerVolumeSpecName "kube-api-access-dbf8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:15.871018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.870942 4927 generic.go:356] "Generic (PLEG): container finished" podID="82cb426a-d010-427a-a153-569fee89d6a7" containerID="6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673" exitCode=0 Oct 30 01:24:15.871018 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.870987 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" event={"ID":"82cb426a-d010-427a-a153-569fee89d6a7","Type":"ContainerDied","Data":"6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673"} Oct 30 01:24:15.871211 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.871049 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" event={"ID":"82cb426a-d010-427a-a153-569fee89d6a7","Type":"ContainerDied","Data":"150ad75b1cc12bdab382683b2680cc544f87cca978af9009f97e6be93a83885c"} Oct 30 01:24:15.871211 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.871072 4927 scope.go:122] "RemoveContainer" containerID="6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673" Oct 30 01:24:15.871211 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.870998 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8" Oct 30 01:24:15.872706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.872646 4927 generic.go:356] "Generic (PLEG): container finished" podID="29ab4ce5-416f-4f47-832a-da72d1263042" containerID="06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865" exitCode=0 Oct 30 01:24:15.872706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.872679 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" event={"ID":"29ab4ce5-416f-4f47-832a-da72d1263042","Type":"ContainerDied","Data":"06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865"} Oct 30 01:24:15.872706 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.872696 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" Oct 30 01:24:15.872964 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.872732 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4" event={"ID":"29ab4ce5-416f-4f47-832a-da72d1263042","Type":"ContainerDied","Data":"e8d876cddef7a022365252366930373f24c472ad0e9153586cd45ac38ff3c869"} Oct 30 01:24:15.874551 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.874524 4927 generic.go:356] "Generic (PLEG): container finished" podID="1cd85074-3f4b-4d36-ad24-fa7abc24a367" containerID="c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04" exitCode=0 Oct 30 01:24:15.874691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.874642 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" event={"ID":"1cd85074-3f4b-4d36-ad24-fa7abc24a367","Type":"ContainerDied","Data":"c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04"} Oct 30 01:24:15.874691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.874659 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" Oct 30 01:24:15.874691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.874683 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv" event={"ID":"1cd85074-3f4b-4d36-ad24-fa7abc24a367","Type":"ContainerDied","Data":"8e87fdfa8abd52cdd663857e1e44e11671f0b0895427e4564adf8e17998bed83"} Oct 30 01:24:15.913616 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.913523 4927 scope.go:122] "RemoveContainer" containerID="6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673" Oct 30 01:24:15.914078 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:15.914029 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673\": container with ID starting with 6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673 not found: ID does not exist" containerID="6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673" Oct 30 01:24:15.914186 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.914074 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673"} err="failed to get container status \"6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673\": rpc error: code = NotFound desc = could not find container \"6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673\": container with ID starting with 6e8fec07b884cc01062cf07de7b686b227154e042ba2eaed5e3270670f368673 not found: ID does not exist" Oct 30 01:24:15.914186 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.914098 4927 scope.go:122] "RemoveContainer" containerID="06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865" Oct 30 01:24:15.922798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.922773 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbf8h\" (UniqueName: \"kubernetes.io/projected/1cd85074-3f4b-4d36-ad24-fa7abc24a367-kube-api-access-dbf8h\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:15.926522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.926502 4927 scope.go:122] "RemoveContainer" containerID="06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865" Oct 30 01:24:15.926816 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:15.926792 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865\": container with ID starting with 06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865 not found: ID does not exist" containerID="06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865" Oct 30 01:24:15.926899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.926824 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865"} err="failed to get container status \"06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865\": rpc error: code = NotFound desc = could not find container \"06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865\": container with ID starting with 06180ead08187c456874f644629034047b9a75a315f4f047bf600f1649005865 not found: ID does not exist" Oct 30 01:24:15.926899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.926840 4927 scope.go:122] "RemoveContainer" containerID="c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04" Oct 30 01:24:15.934457 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.934436 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4"] Oct 30 01:24:15.936983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.936961 4927 scope.go:122] "RemoveContainer" containerID="c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04" Oct 30 01:24:15.937247 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:15.937231 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04\": container with ID starting with c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04 not found: ID does not exist" containerID="c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04" Oct 30 01:24:15.937311 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.937254 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04"} err="failed to get container status \"c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04\": rpc error: code = NotFound desc = could not find container \"c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04\": container with ID starting with c404d03b81b56e67b507095f2a6f26c86268fe4f9df8af4b962addccd7465d04 not found: ID does not exist" Oct 30 01:24:15.938972 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.938949 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5f99e7-5d4ddb8c9b-ql2p4"] Oct 30 01:24:15.988001 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.987972 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8"] Oct 30 01:24:15.996746 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:15.996723 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5f99e7-58f6878784-x95g8"] Oct 30 01:24:16.028794 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.028763 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv"] Oct 30 01:24:16.032145 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.032121 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5f99e7-5454df895f-95pgv"] Oct 30 01:24:16.503315 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.503289 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 01:24:16.528702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.528679 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ce4978-b441-48c9-9145-24c6bdf788f6-kube-api-access-pb4c2\" (UniqueName: \"kubernetes.io/projected/65ce4978-b441-48c9-9145-24c6bdf788f6-kube-api-access-pb4c2\") pod \"65ce4978-b441-48c9-9145-24c6bdf788f6\" (UID: \"65ce4978-b441-48c9-9145-24c6bdf788f6\") " Oct 30 01:24:16.528857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.528779 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ce4978-b441-48c9-9145-24c6bdf788f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ce4978-b441-48c9-9145-24c6bdf788f6-plugins-config-volume\") pod \"65ce4978-b441-48c9-9145-24c6bdf788f6\" (UID: \"65ce4978-b441-48c9-9145-24c6bdf788f6\") " Oct 30 01:24:16.529143 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.529118 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ce4978-b441-48c9-9145-24c6bdf788f6-plugins-config-volume" pod "65ce4978-b441-48c9-9145-24c6bdf788f6" (UID: "65ce4978-b441-48c9-9145-24c6bdf788f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:16.529325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.529306 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ce4978-b441-48c9-9145-24c6bdf788f6-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:16.529519 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.529496 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 01:24:16.530942 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.530920 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ce4978-b441-48c9-9145-24c6bdf788f6-kube-api-access-pb4c2" pod "65ce4978-b441-48c9-9145-24c6bdf788f6" (UID: "65ce4978-b441-48c9-9145-24c6bdf788f6"). InnerVolumeSpecName "kube-api-access-pb4c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:16.562625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.562595 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq"] Oct 30 01:24:16.602872 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.602837 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm"] Oct 30 01:24:16.630091 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.630059 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd70c464-1507-414d-96d9-a138e4143281-kube-api-access-hhmfw\" (UniqueName: \"kubernetes.io/projected/cd70c464-1507-414d-96d9-a138e4143281-kube-api-access-hhmfw\") pod \"cd70c464-1507-414d-96d9-a138e4143281\" (UID: \"cd70c464-1507-414d-96d9-a138e4143281\") " Oct 30 01:24:16.630250 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.630151 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd70c464-1507-414d-96d9-a138e4143281-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd70c464-1507-414d-96d9-a138e4143281-plugins-config-volume\") pod \"cd70c464-1507-414d-96d9-a138e4143281\" (UID: \"cd70c464-1507-414d-96d9-a138e4143281\") " Oct 30 01:24:16.630521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.630496 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd70c464-1507-414d-96d9-a138e4143281-plugins-config-volume" pod "cd70c464-1507-414d-96d9-a138e4143281" (UID: "cd70c464-1507-414d-96d9-a138e4143281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:16.631493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.631470 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4c2\" (UniqueName: \"kubernetes.io/projected/65ce4978-b441-48c9-9145-24c6bdf788f6-kube-api-access-pb4c2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:16.631639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.631495 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd70c464-1507-414d-96d9-a138e4143281-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:16.639394 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.639360 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd70c464-1507-414d-96d9-a138e4143281-kube-api-access-hhmfw" pod "cd70c464-1507-414d-96d9-a138e4143281" (UID: "cd70c464-1507-414d-96d9-a138e4143281"). InnerVolumeSpecName "kube-api-access-hhmfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:16.732394 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.732310 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhmfw\" (UniqueName: \"kubernetes.io/projected/cd70c464-1507-414d-96d9-a138e4143281-kube-api-access-hhmfw\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:16.883987 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.883952 4927 generic.go:356] "Generic (PLEG): container finished" podID="65ce4978-b441-48c9-9145-24c6bdf788f6" containerID="ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7" exitCode=0 Oct 30 01:24:16.884185 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.884009 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" Oct 30 01:24:16.884185 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.884030 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" event={"ID":"65ce4978-b441-48c9-9145-24c6bdf788f6","Type":"ContainerDied","Data":"ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7"} Oct 30 01:24:16.884185 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.884124 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq" event={"ID":"65ce4978-b441-48c9-9145-24c6bdf788f6","Type":"ContainerDied","Data":"6dd5bee9c21dacc0a7174542ba0c411d725cbdc73fdbd340c72efcb8a92da78a"} Oct 30 01:24:16.884185 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.884157 4927 scope.go:122] "RemoveContainer" containerID="ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7" Oct 30 01:24:16.887957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.887932 4927 generic.go:356] "Generic (PLEG): container finished" podID="cd70c464-1507-414d-96d9-a138e4143281" containerID="bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985" exitCode=0 Oct 30 01:24:16.888106 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.888010 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" Oct 30 01:24:16.888106 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.888030 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" event={"ID":"cd70c464-1507-414d-96d9-a138e4143281","Type":"ContainerDied","Data":"bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985"} Oct 30 01:24:16.888106 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.888080 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c" event={"ID":"cd70c464-1507-414d-96d9-a138e4143281","Type":"ContainerDied","Data":"7f87d4cf6131302bbd360abedfc186d2ec801a1af368d85993274ac72b1b284f"} Oct 30 01:24:16.896415 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.896397 4927 scope.go:122] "RemoveContainer" containerID="ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7" Oct 30 01:24:16.896735 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:16.896715 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7\": container with ID starting with ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7 not found: ID does not exist" containerID="ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7" Oct 30 01:24:16.896796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.896744 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7"} err="failed to get container status \"ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7\": rpc error: code = NotFound desc = could not find container \"ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7\": container with ID starting with ef8d9f688affa57de4d544e0ba7210a017da3e4c8d2d7ec529f2afd0dd3b01f7 not found: ID does not exist" Oct 30 01:24:16.896796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.896761 4927 scope.go:122] "RemoveContainer" containerID="bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985" Oct 30 01:24:16.907666 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.907639 4927 scope.go:122] "RemoveContainer" containerID="bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985" Oct 30 01:24:16.907997 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:16.907975 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985\": container with ID starting with bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985 not found: ID does not exist" containerID="bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985" Oct 30 01:24:16.908087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.908005 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985"} err="failed to get container status \"bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985\": rpc error: code = NotFound desc = could not find container \"bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985\": container with ID starting with bfc8bd2944153f6152b781860084e7648e3a4dd6b9a0228e83a1959b1ece1985 not found: ID does not exist" Oct 30 01:24:16.950660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.950627 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq"] Oct 30 01:24:16.956304 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.956276 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-40ac41-776bf85c68-r5qdq"] Oct 30 01:24:16.989775 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.989715 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c"] Oct 30 01:24:16.996494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:16.996472 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-40ac41-86d7dfdfb7-4hn8c"] Oct 30 01:24:17.321122 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.321033 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz"] Oct 30 01:24:17.321329 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.321301 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" podUID="269e1e40-2de3-4384-8d31-282bfb066435" containerName="maas-api" containerID="cri-o://dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1" gracePeriod=30 Oct 30 01:24:17.597340 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.597258 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd85074-3f4b-4d36-ad24-fa7abc24a367" path="/var/lib/kubelet/pods/1cd85074-3f4b-4d36-ad24-fa7abc24a367/volumes" Oct 30 01:24:17.598542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.598515 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ab4ce5-416f-4f47-832a-da72d1263042" path="/var/lib/kubelet/pods/29ab4ce5-416f-4f47-832a-da72d1263042/volumes" Oct 30 01:24:17.599338 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.599231 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ce4978-b441-48c9-9145-24c6bdf788f6" path="/var/lib/kubelet/pods/65ce4978-b441-48c9-9145-24c6bdf788f6/volumes" Oct 30 01:24:17.600402 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.600382 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82cb426a-d010-427a-a153-569fee89d6a7" path="/var/lib/kubelet/pods/82cb426a-d010-427a-a153-569fee89d6a7/volumes" Oct 30 01:24:17.601057 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.601040 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd70c464-1507-414d-96d9-a138e4143281" path="/var/lib/kubelet/pods/cd70c464-1507-414d-96d9-a138e4143281/volumes" Oct 30 01:24:17.656151 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.656126 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 01:24:17.843528 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.843493 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-maas-api-tls\") pod \"269e1e40-2de3-4384-8d31-282bfb066435\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " Oct 30 01:24:17.843734 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.843683 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269e1e40-2de3-4384-8d31-282bfb066435-kube-api-access-tkg8r\" (UniqueName: \"kubernetes.io/projected/269e1e40-2de3-4384-8d31-282bfb066435-kube-api-access-tkg8r\") pod \"269e1e40-2de3-4384-8d31-282bfb066435\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " Oct 30 01:24:17.843823 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.843746 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-metrics-certs\") pod \"269e1e40-2de3-4384-8d31-282bfb066435\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " Oct 30 01:24:17.843823 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.843814 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269e1e40-2de3-4384-8d31-282bfb066435-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e1e40-2de3-4384-8d31-282bfb066435-openshift-service-ca\") pod \"269e1e40-2de3-4384-8d31-282bfb066435\" (UID: \"269e1e40-2de3-4384-8d31-282bfb066435\") " Oct 30 01:24:17.844186 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.844158 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269e1e40-2de3-4384-8d31-282bfb066435-openshift-service-ca" pod "269e1e40-2de3-4384-8d31-282bfb066435" (UID: "269e1e40-2de3-4384-8d31-282bfb066435"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:17.844336 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.844298 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e1e40-2de3-4384-8d31-282bfb066435-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:17.845836 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.845807 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269e1e40-2de3-4384-8d31-282bfb066435-kube-api-access-tkg8r" pod "269e1e40-2de3-4384-8d31-282bfb066435" (UID: "269e1e40-2de3-4384-8d31-282bfb066435"). InnerVolumeSpecName "kube-api-access-tkg8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:17.845947 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.845859 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-metrics-certs" pod "269e1e40-2de3-4384-8d31-282bfb066435" (UID: "269e1e40-2de3-4384-8d31-282bfb066435"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:17.846061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.846046 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-maas-api-tls" pod "269e1e40-2de3-4384-8d31-282bfb066435" (UID: "269e1e40-2de3-4384-8d31-282bfb066435"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:17.901268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.901237 4927 generic.go:356] "Generic (PLEG): container finished" podID="269e1e40-2de3-4384-8d31-282bfb066435" containerID="dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1" exitCode=0 Oct 30 01:24:17.901424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.901297 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" Oct 30 01:24:17.901424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.901323 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" event={"ID":"269e1e40-2de3-4384-8d31-282bfb066435","Type":"ContainerDied","Data":"dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1"} Oct 30 01:24:17.901424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.901378 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz" event={"ID":"269e1e40-2de3-4384-8d31-282bfb066435","Type":"ContainerDied","Data":"ac817ed143143b7084fcb7ccd3c04a2f89919e3395701f86a57325261d8de5aa"} Oct 30 01:24:17.901424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.901401 4927 scope.go:122] "RemoveContainer" containerID="dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1" Oct 30 01:24:17.913525 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.913510 4927 scope.go:122] "RemoveContainer" containerID="dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1" Oct 30 01:24:17.913785 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:17.913768 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1\": container with ID starting with dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1 not found: ID does not exist" containerID="dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1" Oct 30 01:24:17.913848 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.913795 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1"} err="failed to get container status \"dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1\": rpc error: code = NotFound desc = could not find container \"dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1\": container with ID starting with dd895e618fcfe87f4aa7ff6cc576909f729b25f8e9f38589514c54086c9843e1 not found: ID does not exist" Oct 30 01:24:17.945608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.945565 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:17.945608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.945607 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkg8r\" (UniqueName: \"kubernetes.io/projected/269e1e40-2de3-4384-8d31-282bfb066435-kube-api-access-tkg8r\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:17.945801 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.945619 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e1e40-2de3-4384-8d31-282bfb066435-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:17.952683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.952649 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz"] Oct 30 01:24:17.961960 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:17.961935 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-40ac41-647c669dcc-zntmz"] Oct 30 01:24:19.574107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:19.574067 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269e1e40-2de3-4384-8d31-282bfb066435" path="/var/lib/kubelet/pods/269e1e40-2de3-4384-8d31-282bfb066435/volumes" Oct 30 01:24:21.563752 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:21.563707 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" podUID="dbaf19a5-5ea9-4353-80e8-333668ef5bde" containerName="payload-processing" containerID="cri-o://3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527" gracePeriod=25 Oct 30 01:24:21.604107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:21.604064 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" podUID="ba09d4f5-5e04-4733-8e95-65527ab7ca15" containerName="payload-pre-processing" containerID="cri-o://d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a" gracePeriod=25 Oct 30 01:24:22.581456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:22.581420 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:24:22.611202 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:22.611172 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:24:24.227361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:24.227322 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c"] Oct 30 01:24:24.227885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:24.227609 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerName="istio-proxy" containerID="cri-o://e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0" gracePeriod=30 Oct 30 01:24:26.379172 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:26.379131 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" podUID="dbaf19a5-5ea9-4353-80e8-333668ef5bde" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:26.379627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:26.379195 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" podUID="ba09d4f5-5e04-4733-8e95-65527ab7ca15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:26.590725 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:26.590673 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm"] Oct 30 01:24:26.591501 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:26.591269 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" containerID="cri-o://6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b" gracePeriod=30 Oct 30 01:24:29.501385 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.501317 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:24:29.591984 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.591943 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-certs\") pod \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " Oct 30 01:24:29.592159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.591998 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-data\") pod \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " Oct 30 01:24:29.592159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.592036 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-kube-api-access-4xc2m\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-kube-api-access-4xc2m\") pod \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " Oct 30 01:24:29.592159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.592101 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-socket\") pod \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " Oct 30 01:24:29.592159 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.592156 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istiod-ca-cert\") pod \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " Oct 30 01:24:29.592410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.592242 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-certs" pod "2a7e7ec9-3153-438f-820b-c3e8e35bd040" (UID: "2a7e7ec9-3153-438f-820b-c3e8e35bd040"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.592410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.592338 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-data" pod "2a7e7ec9-3153-438f-820b-c3e8e35bd040" (UID: "2a7e7ec9-3153-438f-820b-c3e8e35bd040"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.592531 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.592463 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-envoy\") pod \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " Oct 30 01:24:29.593255 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.592531 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-token\") pod \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " Oct 30 01:24:29.595981 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.595821 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-kube-api-access-4xc2m" pod "2a7e7ec9-3153-438f-820b-c3e8e35bd040" (UID: "2a7e7ec9-3153-438f-820b-c3e8e35bd040"). InnerVolumeSpecName "kube-api-access-4xc2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:29.595981 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.595852 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-credential-socket\") pod \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " Oct 30 01:24:29.596411 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.596012 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-podinfo\") pod \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\" (UID: \"2a7e7ec9-3153-438f-820b-c3e8e35bd040\") " Oct 30 01:24:29.597128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.597101 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-credential-socket" pod "2a7e7ec9-3153-438f-820b-c3e8e35bd040" (UID: "2a7e7ec9-3153-438f-820b-c3e8e35bd040"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.597236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.597124 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.597236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.597149 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.597236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.597163 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xc2m\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-kube-api-access-4xc2m\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.597236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.597177 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-socket" pod "2a7e7ec9-3153-438f-820b-c3e8e35bd040" (UID: "2a7e7ec9-3153-438f-820b-c3e8e35bd040"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.597619 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.597561 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istiod-ca-cert" pod "2a7e7ec9-3153-438f-820b-c3e8e35bd040" (UID: "2a7e7ec9-3153-438f-820b-c3e8e35bd040"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:29.601615 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.599960 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-token" pod "2a7e7ec9-3153-438f-820b-c3e8e35bd040" (UID: "2a7e7ec9-3153-438f-820b-c3e8e35bd040"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:29.618195 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.618164 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-envoy" pod "2a7e7ec9-3153-438f-820b-c3e8e35bd040" (UID: "2a7e7ec9-3153-438f-820b-c3e8e35bd040"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:29.619443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.619414 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-podinfo" pod "2a7e7ec9-3153-438f-820b-c3e8e35bd040" (UID: "2a7e7ec9-3153-438f-820b-c3e8e35bd040"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:29.698318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.698282 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.698318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.698319 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.698564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.698334 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.698564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.698347 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.698564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.698359 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.698564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.698439 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7e7ec9-3153-438f-820b-c3e8e35bd040-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:29.990242 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.990195 4927 generic.go:356] "Generic (PLEG): container finished" podID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerID="e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0" exitCode=0 Oct 30 01:24:29.990438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.990270 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" Oct 30 01:24:29.990438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.990287 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" event={"ID":"2a7e7ec9-3153-438f-820b-c3e8e35bd040","Type":"ContainerDied","Data":"e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0"} Oct 30 01:24:29.990438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.990373 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c" event={"ID":"2a7e7ec9-3153-438f-820b-c3e8e35bd040","Type":"ContainerDied","Data":"90cd243f270294c5e35c3f5229cead43dea8e34901d238111a22b1d611f0a755"} Oct 30 01:24:29.990438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:29.990399 4927 scope.go:122] "RemoveContainer" containerID="e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0" Oct 30 01:24:30.001486 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:30.001465 4927 scope.go:122] "RemoveContainer" containerID="e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0" Oct 30 01:24:30.001785 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:30.001766 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0\": container with ID starting with e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0 not found: ID does not exist" containerID="e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0" Oct 30 01:24:30.001860 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:30.001795 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0"} err="failed to get container status \"e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0\": rpc error: code = NotFound desc = could not find container \"e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0\": container with ID starting with e7baff9c7b71ac181e3f8eeb545d910c9f573e85767059030221fae638fa97e0 not found: ID does not exist" Oct 30 01:24:30.040760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:30.040733 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c"] Oct 30 01:24:30.051719 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:30.051695 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5f99e7-openshift-default-65d869d4b9-jc28c"] Oct 30 01:24:31.571805 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.571770 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" path="/var/lib/kubelet/pods/2a7e7ec9-3153-438f-820b-c3e8e35bd040/volumes" Oct 30 01:24:31.861736 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.861709 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 01:24:31.924070 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924030 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-podinfo\") pod \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " Oct 30 01:24:31.924263 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924121 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-certs\") pod \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " Oct 30 01:24:31.924263 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924185 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-kube-api-access-nllnp\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-kube-api-access-nllnp\") pod \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " Oct 30 01:24:31.924263 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924236 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-data\") pod \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " Oct 30 01:24:31.924441 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924271 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-token\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-token\") pod \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " Oct 30 01:24:31.924441 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924297 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-credential-socket\") pod \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " Oct 30 01:24:31.924441 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924341 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istiod-ca-cert\") pod \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " Oct 30 01:24:31.924627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924444 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-socket\") pod \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " Oct 30 01:24:31.924627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924545 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-envoy\") pod \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\" (UID: \"e94498ce-f9f6-4586-8324-e1c5cd85c4f4\") " Oct 30 01:24:31.924776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924731 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-credential-socket" pod "e94498ce-f9f6-4586-8324-e1c5cd85c4f4" (UID: "e94498ce-f9f6-4586-8324-e1c5cd85c4f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.924838 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924786 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-certs" pod "e94498ce-f9f6-4586-8324-e1c5cd85c4f4" (UID: "e94498ce-f9f6-4586-8324-e1c5cd85c4f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.925130 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.925014 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-data" pod "e94498ce-f9f6-4586-8324-e1c5cd85c4f4" (UID: "e94498ce-f9f6-4586-8324-e1c5cd85c4f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.925130 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.924937 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istiod-ca-cert" pod "e94498ce-f9f6-4586-8324-e1c5cd85c4f4" (UID: "e94498ce-f9f6-4586-8324-e1c5cd85c4f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:31.925130 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.925088 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-socket" pod "e94498ce-f9f6-4586-8324-e1c5cd85c4f4" (UID: "e94498ce-f9f6-4586-8324-e1c5cd85c4f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.925400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.925177 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.925400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.925200 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.925400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.925213 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.925400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.925229 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.928186 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.927231 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-podinfo" pod "e94498ce-f9f6-4586-8324-e1c5cd85c4f4" (UID: "e94498ce-f9f6-4586-8324-e1c5cd85c4f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:31.928186 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.927351 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-envoy" pod "e94498ce-f9f6-4586-8324-e1c5cd85c4f4" (UID: "e94498ce-f9f6-4586-8324-e1c5cd85c4f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:31.928186 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.927706 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-kube-api-access-nllnp" pod "e94498ce-f9f6-4586-8324-e1c5cd85c4f4" (UID: "e94498ce-f9f6-4586-8324-e1c5cd85c4f4"). InnerVolumeSpecName "kube-api-access-nllnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:31.928927 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:31.928750 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-token" pod "e94498ce-f9f6-4586-8324-e1c5cd85c4f4" (UID: "e94498ce-f9f6-4586-8324-e1c5cd85c4f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:32.005056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.005018 4927 generic.go:356] "Generic (PLEG): container finished" podID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerID="6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b" exitCode=0 Oct 30 01:24:32.005208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.005070 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" Oct 30 01:24:32.005208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.005096 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" event={"ID":"e94498ce-f9f6-4586-8324-e1c5cd85c4f4","Type":"ContainerDied","Data":"6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b"} Oct 30 01:24:32.005208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.005154 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm" event={"ID":"e94498ce-f9f6-4586-8324-e1c5cd85c4f4","Type":"ContainerDied","Data":"279f27c31f8810fc2ab9b69a55031aae645fa64cb0b53cc07a5f1d70513c046d"} Oct 30 01:24:32.005208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.005177 4927 scope.go:122] "RemoveContainer" containerID="6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b" Oct 30 01:24:32.017476 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.017457 4927 scope.go:122] "RemoveContainer" containerID="6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b" Oct 30 01:24:32.017789 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:32.017764 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b\": container with ID starting with 6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b not found: ID does not exist" containerID="6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b" Oct 30 01:24:32.017885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.017799 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b"} err="failed to get container status \"6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b\": rpc error: code = NotFound desc = could not find container \"6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b\": container with ID starting with 6220daa2246a0a610eca1a11fcd0f562ca0f9d1232df70900189840fd79d7a4b not found: ID does not exist" Oct 30 01:24:32.026698 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.026669 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.026698 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.026695 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.026838 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.026710 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.026838 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.026722 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nllnp\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-kube-api-access-nllnp\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.026838 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.026734 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94498ce-f9f6-4586-8324-e1c5cd85c4f4-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.056172 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.056145 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm"] Oct 30 01:24:32.060457 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:32.060435 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tqxtm"] Oct 30 01:24:33.571771 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:33.571682 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" path="/var/lib/kubelet/pods/e94498ce-f9f6-4586-8324-e1c5cd85c4f4/volumes" Oct 30 01:24:34.442249 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:34.442214 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:24:36.379517 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:36.379477 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" podUID="dbaf19a5-5ea9-4353-80e8-333668ef5bde" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:36.379924 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:36.379520 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" podUID="ba09d4f5-5e04-4733-8e95-65527ab7ca15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:39.235882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.235841 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl"] Oct 30 01:24:39.236314 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.236069 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" podUID="febc6e30-896a-495a-af9c-7a6a65e40975" containerName="maas-api" containerID="cri-o://f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270" gracePeriod=30 Oct 30 01:24:39.442473 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.442414 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx"] Oct 30 01:24:39.443274 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.443231 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" podUID="414818dd-4954-4548-be30-01f1a9ded7c5" containerName="payload-pre-processing" containerID="cri-o://3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d" gracePeriod=30 Oct 30 01:24:39.581306 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.581274 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:24:39.691498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.691468 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:24:39.731441 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.731406 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-metrics-certs\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-metrics-certs\") pod \"febc6e30-896a-495a-af9c-7a6a65e40975\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " Oct 30 01:24:39.731658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.731460 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-maas-api-tls\") pod \"febc6e30-896a-495a-af9c-7a6a65e40975\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " Oct 30 01:24:39.731658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.731511 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febc6e30-896a-495a-af9c-7a6a65e40975-kube-api-access-9jq2g\" (UniqueName: \"kubernetes.io/projected/febc6e30-896a-495a-af9c-7a6a65e40975-kube-api-access-9jq2g\") pod \"febc6e30-896a-495a-af9c-7a6a65e40975\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " Oct 30 01:24:39.731658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.731623 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/febc6e30-896a-495a-af9c-7a6a65e40975-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febc6e30-896a-495a-af9c-7a6a65e40975-openshift-service-ca\") pod \"febc6e30-896a-495a-af9c-7a6a65e40975\" (UID: \"febc6e30-896a-495a-af9c-7a6a65e40975\") " Oct 30 01:24:39.732526 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.732170 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/febc6e30-896a-495a-af9c-7a6a65e40975-openshift-service-ca" pod "febc6e30-896a-495a-af9c-7a6a65e40975" (UID: "febc6e30-896a-495a-af9c-7a6a65e40975"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:39.735093 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.734835 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febc6e30-896a-495a-af9c-7a6a65e40975-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:39.735646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.735605 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febc6e30-896a-495a-af9c-7a6a65e40975-kube-api-access-9jq2g" pod "febc6e30-896a-495a-af9c-7a6a65e40975" (UID: "febc6e30-896a-495a-af9c-7a6a65e40975"). InnerVolumeSpecName "kube-api-access-9jq2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:39.737154 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.737128 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-metrics-certs" pod "febc6e30-896a-495a-af9c-7a6a65e40975" (UID: "febc6e30-896a-495a-af9c-7a6a65e40975"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:39.737154 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.737140 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-maas-api-tls" pod "febc6e30-896a-495a-af9c-7a6a65e40975" (UID: "febc6e30-896a-495a-af9c-7a6a65e40975"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:39.835542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.835445 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414818dd-4954-4548-be30-01f1a9ded7c5-kube-api-access-crpjf\" (UniqueName: \"kubernetes.io/projected/414818dd-4954-4548-be30-01f1a9ded7c5-kube-api-access-crpjf\") pod \"414818dd-4954-4548-be30-01f1a9ded7c5\" (UID: \"414818dd-4954-4548-be30-01f1a9ded7c5\") " Oct 30 01:24:39.835717 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.835697 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414818dd-4954-4548-be30-01f1a9ded7c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414818dd-4954-4548-be30-01f1a9ded7c5-plugins-config-volume\") pod \"414818dd-4954-4548-be30-01f1a9ded7c5\" (UID: \"414818dd-4954-4548-be30-01f1a9ded7c5\") " Oct 30 01:24:39.836079 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.836059 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:39.836079 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.836079 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febc6e30-896a-495a-af9c-7a6a65e40975-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:39.836288 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.836093 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jq2g\" (UniqueName: \"kubernetes.io/projected/febc6e30-896a-495a-af9c-7a6a65e40975-kube-api-access-9jq2g\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:39.836288 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.836144 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414818dd-4954-4548-be30-01f1a9ded7c5-plugins-config-volume" pod "414818dd-4954-4548-be30-01f1a9ded7c5" (UID: "414818dd-4954-4548-be30-01f1a9ded7c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:39.838340 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.838310 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414818dd-4954-4548-be30-01f1a9ded7c5-kube-api-access-crpjf" pod "414818dd-4954-4548-be30-01f1a9ded7c5" (UID: "414818dd-4954-4548-be30-01f1a9ded7c5"). InnerVolumeSpecName "kube-api-access-crpjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:39.937322 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.937280 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414818dd-4954-4548-be30-01f1a9ded7c5-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:39.937322 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:39.937314 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crpjf\" (UniqueName: \"kubernetes.io/projected/414818dd-4954-4548-be30-01f1a9ded7c5-kube-api-access-crpjf\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:40.075247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.075206 4927 generic.go:356] "Generic (PLEG): container finished" podID="414818dd-4954-4548-be30-01f1a9ded7c5" containerID="3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d" exitCode=0 Oct 30 01:24:40.075468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.075251 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" Oct 30 01:24:40.075468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.075295 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" event={"ID":"414818dd-4954-4548-be30-01f1a9ded7c5","Type":"ContainerDied","Data":"3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d"} Oct 30 01:24:40.075468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.075363 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx" event={"ID":"414818dd-4954-4548-be30-01f1a9ded7c5","Type":"ContainerDied","Data":"512af93ff8ec821283671c06191f7f0ae5b5a5b9507dbccefcb573edc5f41949"} Oct 30 01:24:40.075468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.075393 4927 scope.go:122] "RemoveContainer" containerID="3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d" Oct 30 01:24:40.077055 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.077030 4927 generic.go:356] "Generic (PLEG): container finished" podID="febc6e30-896a-495a-af9c-7a6a65e40975" containerID="f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270" exitCode=0 Oct 30 01:24:40.077184 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.077089 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" Oct 30 01:24:40.077184 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.077143 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" event={"ID":"febc6e30-896a-495a-af9c-7a6a65e40975","Type":"ContainerDied","Data":"f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270"} Oct 30 01:24:40.077321 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.077203 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl" event={"ID":"febc6e30-896a-495a-af9c-7a6a65e40975","Type":"ContainerDied","Data":"7cedf422dafd9718d471eda0c85b60f908e787b1c56e71d376979721363dc26c"} Oct 30 01:24:40.088816 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.088791 4927 scope.go:122] "RemoveContainer" containerID="3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d" Oct 30 01:24:40.089236 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:40.089216 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d\": container with ID starting with 3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d not found: ID does not exist" containerID="3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d" Oct 30 01:24:40.089358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.089249 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d"} err="failed to get container status \"3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d\": rpc error: code = NotFound desc = could not find container \"3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d\": container with ID starting with 3018d82bc27b7a65882f98f26e60ff160b0ada347ed5a5507cb9d2fc4e8f495d not found: ID does not exist" Oct 30 01:24:40.089358 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.089272 4927 scope.go:122] "RemoveContainer" containerID="f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270" Oct 30 01:24:40.101364 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.101335 4927 scope.go:122] "RemoveContainer" containerID="f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270" Oct 30 01:24:40.101677 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:40.101655 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270\": container with ID starting with f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270 not found: ID does not exist" containerID="f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270" Oct 30 01:24:40.101722 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.101685 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270"} err="failed to get container status \"f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270\": rpc error: code = NotFound desc = could not find container \"f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270\": container with ID starting with f9d0a294de196aa7c441b85b75023240380a8eef8fb39fa0382b677d7b103270 not found: ID does not exist" Oct 30 01:24:40.138720 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.138693 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx"] Oct 30 01:24:40.142705 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.142681 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e9921-dd76f5d87-w94vx"] Oct 30 01:24:40.177447 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.177416 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl"] Oct 30 01:24:40.182144 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.182106 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e9921-857d6d5fc7-k56gl"] Oct 30 01:24:40.553771 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.553735 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8"] Oct 30 01:24:40.554246 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:40.554007 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" containerID="cri-o://667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70" gracePeriod=30 Oct 30 01:24:41.578901 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.571710 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414818dd-4954-4548-be30-01f1a9ded7c5" path="/var/lib/kubelet/pods/414818dd-4954-4548-be30-01f1a9ded7c5/volumes" Oct 30 01:24:41.578901 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.572192 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febc6e30-896a-495a-af9c-7a6a65e40975" path="/var/lib/kubelet/pods/febc6e30-896a-495a-af9c-7a6a65e40975/volumes" Oct 30 01:24:41.895495 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.895465 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 01:24:41.925861 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.925834 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 01:24:41.967866 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.967828 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbaf19a5-5ea9-4353-80e8-333668ef5bde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbaf19a5-5ea9-4353-80e8-333668ef5bde-plugins-config-volume\") pod \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\" (UID: \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\") " Oct 30 01:24:41.968056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.967923 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba09d4f5-5e04-4733-8e95-65527ab7ca15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba09d4f5-5e04-4733-8e95-65527ab7ca15-plugins-config-volume\") pod \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\" (UID: \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\") " Oct 30 01:24:41.968130 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.968048 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbaf19a5-5ea9-4353-80e8-333668ef5bde-kube-api-access-pwrp7\" (UniqueName: \"kubernetes.io/projected/dbaf19a5-5ea9-4353-80e8-333668ef5bde-kube-api-access-pwrp7\") pod \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\" (UID: \"dbaf19a5-5ea9-4353-80e8-333668ef5bde\") " Oct 30 01:24:41.968206 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.968188 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba09d4f5-5e04-4733-8e95-65527ab7ca15-kube-api-access-8nt8k\" (UniqueName: \"kubernetes.io/projected/ba09d4f5-5e04-4733-8e95-65527ab7ca15-kube-api-access-8nt8k\") pod \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\" (UID: \"ba09d4f5-5e04-4733-8e95-65527ab7ca15\") " Oct 30 01:24:41.968259 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.968229 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbaf19a5-5ea9-4353-80e8-333668ef5bde-plugins-config-volume" pod "dbaf19a5-5ea9-4353-80e8-333668ef5bde" (UID: "dbaf19a5-5ea9-4353-80e8-333668ef5bde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:41.968397 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.968372 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba09d4f5-5e04-4733-8e95-65527ab7ca15-plugins-config-volume" pod "ba09d4f5-5e04-4733-8e95-65527ab7ca15" (UID: "ba09d4f5-5e04-4733-8e95-65527ab7ca15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:41.968734 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.968705 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba09d4f5-5e04-4733-8e95-65527ab7ca15-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:41.968882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.968738 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbaf19a5-5ea9-4353-80e8-333668ef5bde-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:41.970425 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.970398 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba09d4f5-5e04-4733-8e95-65527ab7ca15-kube-api-access-8nt8k" pod "ba09d4f5-5e04-4733-8e95-65527ab7ca15" (UID: "ba09d4f5-5e04-4733-8e95-65527ab7ca15"). InnerVolumeSpecName "kube-api-access-8nt8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:41.970538 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:41.970441 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbaf19a5-5ea9-4353-80e8-333668ef5bde-kube-api-access-pwrp7" pod "dbaf19a5-5ea9-4353-80e8-333668ef5bde" (UID: "dbaf19a5-5ea9-4353-80e8-333668ef5bde"). InnerVolumeSpecName "kube-api-access-pwrp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:42.069767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.069728 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nt8k\" (UniqueName: \"kubernetes.io/projected/ba09d4f5-5e04-4733-8e95-65527ab7ca15-kube-api-access-8nt8k\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:42.069767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.069769 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwrp7\" (UniqueName: \"kubernetes.io/projected/dbaf19a5-5ea9-4353-80e8-333668ef5bde-kube-api-access-pwrp7\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:42.093521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.093438 4927 generic.go:356] "Generic (PLEG): container finished" podID="ba09d4f5-5e04-4733-8e95-65527ab7ca15" containerID="d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a" exitCode=0 Oct 30 01:24:42.093521 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.093487 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" Oct 30 01:24:42.093778 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.093516 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" event={"ID":"ba09d4f5-5e04-4733-8e95-65527ab7ca15","Type":"ContainerDied","Data":"d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a"} Oct 30 01:24:42.093778 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.093567 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm" event={"ID":"ba09d4f5-5e04-4733-8e95-65527ab7ca15","Type":"ContainerDied","Data":"927c43e87c2f5a24b823d1d654c22eee4a62f72de6dfe461ec4704f9ff003b3d"} Oct 30 01:24:42.093778 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.093599 4927 scope.go:122] "RemoveContainer" containerID="d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a" Oct 30 01:24:42.095198 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.095173 4927 generic.go:356] "Generic (PLEG): container finished" podID="dbaf19a5-5ea9-4353-80e8-333668ef5bde" containerID="3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527" exitCode=0 Oct 30 01:24:42.095318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.095208 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" event={"ID":"dbaf19a5-5ea9-4353-80e8-333668ef5bde","Type":"ContainerDied","Data":"3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527"} Oct 30 01:24:42.095318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.095213 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" Oct 30 01:24:42.095318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.095237 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq" event={"ID":"dbaf19a5-5ea9-4353-80e8-333668ef5bde","Type":"ContainerDied","Data":"a5d03d83ecaed080e0e4d55ff391e70c8ecf5bbc7af7c4f0881be686a19310fa"} Oct 30 01:24:42.105331 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.105314 4927 scope.go:122] "RemoveContainer" containerID="d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a" Oct 30 01:24:42.105629 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:42.105608 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a\": container with ID starting with d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a not found: ID does not exist" containerID="d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a" Oct 30 01:24:42.105702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.105636 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a"} err="failed to get container status \"d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a\": rpc error: code = NotFound desc = could not find container \"d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a\": container with ID starting with d7d6afa480f8e578b02be428accfb2a98026007f6ec6482eabeab004cb1b225a not found: ID does not exist" Oct 30 01:24:42.105702 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.105653 4927 scope.go:122] "RemoveContainer" containerID="3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527" Oct 30 01:24:42.116194 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.116173 4927 scope.go:122] "RemoveContainer" containerID="3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527" Oct 30 01:24:42.116517 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:42.116479 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527\": container with ID starting with 3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527 not found: ID does not exist" containerID="3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527" Oct 30 01:24:42.117349 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.116527 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527"} err="failed to get container status \"3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527\": rpc error: code = NotFound desc = could not find container \"3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527\": container with ID starting with 3d19d1d3902107c93f3869fd7725a08e989ee50801a6713d1b43f1c0094d1527 not found: ID does not exist" Oct 30 01:24:42.142262 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.142240 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq"] Oct 30 01:24:42.146184 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.146164 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5062f6-54896b9c4d-986mq"] Oct 30 01:24:42.177970 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.177940 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm"] Oct 30 01:24:42.183741 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:42.183713 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5062f6-5c47cc4b8c-rgtvm"] Oct 30 01:24:43.570905 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:43.570871 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba09d4f5-5e04-4733-8e95-65527ab7ca15" path="/var/lib/kubelet/pods/ba09d4f5-5e04-4733-8e95-65527ab7ca15/volumes" Oct 30 01:24:43.571414 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:43.571398 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbaf19a5-5ea9-4353-80e8-333668ef5bde" path="/var/lib/kubelet/pods/dbaf19a5-5ea9-4353-80e8-333668ef5bde/volumes" Oct 30 01:24:45.832775 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.832718 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:24:45.914803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.914771 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-credential-socket\") pod \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " Oct 30 01:24:45.914982 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.914808 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istiod-ca-cert\") pod \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " Oct 30 01:24:45.914982 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.914860 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-certs\") pod \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " Oct 30 01:24:45.914982 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.914946 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-socket\") pod \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " Oct 30 01:24:45.915152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915028 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-podinfo\") pod \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " Oct 30 01:24:45.915152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915068 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-kube-api-access-2p4cm\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-kube-api-access-2p4cm\") pod \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " Oct 30 01:24:45.915152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915094 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-data\") pod \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " Oct 30 01:24:45.915152 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915144 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-envoy\") pod \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " Oct 30 01:24:45.915390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915178 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-token\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-token\") pod \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\" (UID: \"28e21533-1c44-41fa-82a3-f5ed8296bdb4\") " Oct 30 01:24:45.915390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915086 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-credential-socket" pod "28e21533-1c44-41fa-82a3-f5ed8296bdb4" (UID: "28e21533-1c44-41fa-82a3-f5ed8296bdb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.915390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915199 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-certs" pod "28e21533-1c44-41fa-82a3-f5ed8296bdb4" (UID: "28e21533-1c44-41fa-82a3-f5ed8296bdb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.915390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915356 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-socket" pod "28e21533-1c44-41fa-82a3-f5ed8296bdb4" (UID: "28e21533-1c44-41fa-82a3-f5ed8296bdb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.915390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915364 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istiod-ca-cert" pod "28e21533-1c44-41fa-82a3-f5ed8296bdb4" (UID: "28e21533-1c44-41fa-82a3-f5ed8296bdb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:45.915666 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915646 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.915728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915670 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.915728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915682 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.915728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915694 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.915890 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.915797 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-data" pod "28e21533-1c44-41fa-82a3-f5ed8296bdb4" (UID: "28e21533-1c44-41fa-82a3-f5ed8296bdb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.918014 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.917982 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-token" pod "28e21533-1c44-41fa-82a3-f5ed8296bdb4" (UID: "28e21533-1c44-41fa-82a3-f5ed8296bdb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:45.918170 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.918017 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-podinfo" pod "28e21533-1c44-41fa-82a3-f5ed8296bdb4" (UID: "28e21533-1c44-41fa-82a3-f5ed8296bdb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:45.918479 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.918456 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-envoy" pod "28e21533-1c44-41fa-82a3-f5ed8296bdb4" (UID: "28e21533-1c44-41fa-82a3-f5ed8296bdb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.919741 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:45.919687 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-kube-api-access-2p4cm" pod "28e21533-1c44-41fa-82a3-f5ed8296bdb4" (UID: "28e21533-1c44-41fa-82a3-f5ed8296bdb4"). InnerVolumeSpecName "kube-api-access-2p4cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:46.016555 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.016515 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:46.016555 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.016547 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p4cm\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-kube-api-access-2p4cm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:46.016555 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.016559 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:46.016817 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.016600 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:46.016817 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.016614 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e21533-1c44-41fa-82a3-f5ed8296bdb4-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:46.158215 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.158179 4927 generic.go:356] "Generic (PLEG): container finished" podID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerID="667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70" exitCode=0 Oct 30 01:24:46.158416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.158232 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" Oct 30 01:24:46.158416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.158262 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" event={"ID":"28e21533-1c44-41fa-82a3-f5ed8296bdb4","Type":"ContainerDied","Data":"667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70"} Oct 30 01:24:46.158416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.158323 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8" event={"ID":"28e21533-1c44-41fa-82a3-f5ed8296bdb4","Type":"ContainerDied","Data":"9c25e38f9d3b68633b0dec6a946f030ecf841005c5ce27e1de0ab272977b0cf1"} Oct 30 01:24:46.158416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.158355 4927 scope.go:122] "RemoveContainer" containerID="667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70" Oct 30 01:24:46.169771 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.169748 4927 scope.go:122] "RemoveContainer" containerID="7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74" Oct 30 01:24:46.179547 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.179522 4927 scope.go:122] "RemoveContainer" containerID="667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70" Oct 30 01:24:46.179864 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:46.179844 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70\": container with ID starting with 667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70 not found: ID does not exist" containerID="667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70" Oct 30 01:24:46.179916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.179873 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70"} err="failed to get container status \"667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70\": rpc error: code = NotFound desc = could not find container \"667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70\": container with ID starting with 667497c5f89060858bdf5c78511329b79cbe1e26527facf214c53b5603ec7e70 not found: ID does not exist" Oct 30 01:24:46.179916 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.179892 4927 scope.go:122] "RemoveContainer" containerID="7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74" Oct 30 01:24:46.180129 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:46.180107 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74\": container with ID starting with 7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74 not found: ID does not exist" containerID="7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74" Oct 30 01:24:46.180194 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.180138 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74"} err="failed to get container status \"7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74\": rpc error: code = NotFound desc = could not find container \"7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74\": container with ID starting with 7b1768bcce5396285957b19e9e3f6700b0923bc1a38d6fc66946513a3b233d74 not found: ID does not exist" Oct 30 01:24:46.218965 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.218933 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8"] Oct 30 01:24:46.222443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:46.222419 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-abf8ae-openshift-default-578749ddbf-nqnr8"] Oct 30 01:24:47.572179 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:47.572142 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" path="/var/lib/kubelet/pods/28e21533-1c44-41fa-82a3-f5ed8296bdb4/volumes" Oct 30 01:24:48.210396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:48.210362 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7"] Oct 30 01:24:48.210627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:48.210562 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" containerID="cri-o://79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1" gracePeriod=30 Oct 30 01:24:50.237611 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.237560 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598"] Oct 30 01:24:50.238060 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.237766 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" podUID="a8d33d24-b5e3-4e1d-b039-b14a8ca03432" containerName="maas-api" containerID="cri-o://ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded" gracePeriod=30 Oct 30 01:24:50.531764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.531739 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 01:24:50.574497 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.574459 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-kube-api-access-7dwc7\" (UniqueName: \"kubernetes.io/projected/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-kube-api-access-7dwc7\") pod \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " Oct 30 01:24:50.574731 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.574557 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls\") pod \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " Oct 30 01:24:50.574731 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.574633 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-metrics-certs\") pod \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " Oct 30 01:24:50.574731 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.574662 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-openshift-service-ca\") pod \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\" (UID: \"a8d33d24-b5e3-4e1d-b039-b14a8ca03432\") " Oct 30 01:24:50.575105 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.575071 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-openshift-service-ca" pod "a8d33d24-b5e3-4e1d-b039-b14a8ca03432" (UID: "a8d33d24-b5e3-4e1d-b039-b14a8ca03432"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:50.575260 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.575239 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:50.577078 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.577054 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-kube-api-access-7dwc7" pod "a8d33d24-b5e3-4e1d-b039-b14a8ca03432" (UID: "a8d33d24-b5e3-4e1d-b039-b14a8ca03432"). InnerVolumeSpecName "kube-api-access-7dwc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:50.577335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.577313 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-metrics-certs" pod "a8d33d24-b5e3-4e1d-b039-b14a8ca03432" (UID: "a8d33d24-b5e3-4e1d-b039-b14a8ca03432"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:50.577436 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.577332 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls" pod "a8d33d24-b5e3-4e1d-b039-b14a8ca03432" (UID: "a8d33d24-b5e3-4e1d-b039-b14a8ca03432"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:50.675922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.675880 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:50.675922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.675910 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:50.675922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:50.675921 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dwc7\" (UniqueName: \"kubernetes.io/projected/a8d33d24-b5e3-4e1d-b039-b14a8ca03432-kube-api-access-7dwc7\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:51.202069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.197072 4927 generic.go:356] "Generic (PLEG): container finished" podID="a8d33d24-b5e3-4e1d-b039-b14a8ca03432" containerID="ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded" exitCode=0 Oct 30 01:24:51.202069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.197288 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" event={"ID":"a8d33d24-b5e3-4e1d-b039-b14a8ca03432","Type":"ContainerDied","Data":"ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded"} Oct 30 01:24:51.202069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.197372 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" event={"ID":"a8d33d24-b5e3-4e1d-b039-b14a8ca03432","Type":"ContainerDied","Data":"4234de34b4ea0c2559a1beb4dd1239c50e44de7571e60adaa7b5e19900ceca1d"} Oct 30 01:24:51.202069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.197404 4927 scope.go:122] "RemoveContainer" containerID="ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded" Oct 30 01:24:51.202069 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.197559 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598" Oct 30 01:24:51.226683 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.226617 4927 scope.go:122] "RemoveContainer" containerID="ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded" Oct 30 01:24:51.227048 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:51.227015 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded\": container with ID starting with ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded not found: ID does not exist" containerID="ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded" Oct 30 01:24:51.227162 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.227060 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded"} err="failed to get container status \"ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded\": rpc error: code = NotFound desc = could not find container \"ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded\": container with ID starting with ba926abd244319ceb88d054c7e948fb39b835f809eb98abda82e9a0b67bd9ded not found: ID does not exist" Oct 30 01:24:51.257884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.257850 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598"] Oct 30 01:24:51.268874 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.268846 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5062f6-7dc4866cf9-98598"] Oct 30 01:24:51.571691 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:51.571615 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d33d24-b5e3-4e1d-b039-b14a8ca03432" path="/var/lib/kubelet/pods/a8d33d24-b5e3-4e1d-b039-b14a8ca03432/volumes" Oct 30 01:24:53.493648 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.493559 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:24:53.609405 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609368 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-kube-api-access-qptcd\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-kube-api-access-qptcd\") pod \"4836cefc-d5ca-469d-899b-d9298b44db87\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " Oct 30 01:24:53.609646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609418 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-credential-socket\") pod \"4836cefc-d5ca-469d-899b-d9298b44db87\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " Oct 30 01:24:53.609646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609451 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4836cefc-d5ca-469d-899b-d9298b44db87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4836cefc-d5ca-469d-899b-d9298b44db87-istiod-ca-cert\") pod \"4836cefc-d5ca-469d-899b-d9298b44db87\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " Oct 30 01:24:53.609646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609479 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-certs\") pod \"4836cefc-d5ca-469d-899b-d9298b44db87\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " Oct 30 01:24:53.609646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609622 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-socket\") pod \"4836cefc-d5ca-469d-899b-d9298b44db87\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " Oct 30 01:24:53.609889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609673 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-envoy\") pod \"4836cefc-d5ca-469d-899b-d9298b44db87\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " Oct 30 01:24:53.609889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609752 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-istio-token\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-istio-token\") pod \"4836cefc-d5ca-469d-899b-d9298b44db87\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " Oct 30 01:24:53.609889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609809 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4836cefc-d5ca-469d-899b-d9298b44db87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4836cefc-d5ca-469d-899b-d9298b44db87-istio-podinfo\") pod \"4836cefc-d5ca-469d-899b-d9298b44db87\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " Oct 30 01:24:53.609889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609805 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-credential-socket" pod "4836cefc-d5ca-469d-899b-d9298b44db87" (UID: "4836cefc-d5ca-469d-899b-d9298b44db87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:53.609889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609836 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-certs" pod "4836cefc-d5ca-469d-899b-d9298b44db87" (UID: "4836cefc-d5ca-469d-899b-d9298b44db87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:53.609889 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609869 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-data\") pod \"4836cefc-d5ca-469d-899b-d9298b44db87\" (UID: \"4836cefc-d5ca-469d-899b-d9298b44db87\") " Oct 30 01:24:53.610182 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609897 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4836cefc-d5ca-469d-899b-d9298b44db87-istiod-ca-cert" pod "4836cefc-d5ca-469d-899b-d9298b44db87" (UID: "4836cefc-d5ca-469d-899b-d9298b44db87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:53.610182 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.609998 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-socket" pod "4836cefc-d5ca-469d-899b-d9298b44db87" (UID: "4836cefc-d5ca-469d-899b-d9298b44db87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:53.610339 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.610230 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-data" pod "4836cefc-d5ca-469d-899b-d9298b44db87" (UID: "4836cefc-d5ca-469d-899b-d9298b44db87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:53.610408 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.610388 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:53.610456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.610415 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4836cefc-d5ca-469d-899b-d9298b44db87-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:53.610456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.610432 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:53.610456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.610445 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:53.610628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.610459 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:53.612201 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.612161 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-envoy" pod "4836cefc-d5ca-469d-899b-d9298b44db87" (UID: "4836cefc-d5ca-469d-899b-d9298b44db87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:53.612378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.612276 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-kube-api-access-qptcd" pod "4836cefc-d5ca-469d-899b-d9298b44db87" (UID: "4836cefc-d5ca-469d-899b-d9298b44db87"). InnerVolumeSpecName "kube-api-access-qptcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:53.612451 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.612409 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4836cefc-d5ca-469d-899b-d9298b44db87-istio-podinfo" pod "4836cefc-d5ca-469d-899b-d9298b44db87" (UID: "4836cefc-d5ca-469d-899b-d9298b44db87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:53.612515 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.612479 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-istio-token" pod "4836cefc-d5ca-469d-899b-d9298b44db87" (UID: "4836cefc-d5ca-469d-899b-d9298b44db87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:53.711562 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.711528 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4836cefc-d5ca-469d-899b-d9298b44db87-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:53.711562 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.711554 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qptcd\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-kube-api-access-qptcd\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:53.711785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.711564 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4836cefc-d5ca-469d-899b-d9298b44db87-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:53.711785 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:53.711598 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4836cefc-d5ca-469d-899b-d9298b44db87-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:24:54.221661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.221624 4927 generic.go:356] "Generic (PLEG): container finished" podID="4836cefc-d5ca-469d-899b-d9298b44db87" containerID="79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1" exitCode=0 Oct 30 01:24:54.221894 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.221691 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" Oct 30 01:24:54.221894 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.221719 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" event={"ID":"4836cefc-d5ca-469d-899b-d9298b44db87","Type":"ContainerDied","Data":"79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1"} Oct 30 01:24:54.221894 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.221816 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7" event={"ID":"4836cefc-d5ca-469d-899b-d9298b44db87","Type":"ContainerDied","Data":"8e82ae47ca00b12528df16e90f1c12a190b4a73d02d940ebe8bd72971b04fb2c"} Oct 30 01:24:54.221894 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.221847 4927 scope.go:122] "RemoveContainer" containerID="79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1" Oct 30 01:24:54.233349 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.233323 4927 scope.go:122] "RemoveContainer" containerID="3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426" Oct 30 01:24:54.242707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.242686 4927 scope.go:122] "RemoveContainer" containerID="79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1" Oct 30 01:24:54.242959 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:54.242937 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1\": container with ID starting with 79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1 not found: ID does not exist" containerID="79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1" Oct 30 01:24:54.243008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.242969 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1"} err="failed to get container status \"79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1\": rpc error: code = NotFound desc = could not find container \"79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1\": container with ID starting with 79e95c306197671b90e380c70198e3731524e3a870adc895cea73faccaa010d1 not found: ID does not exist" Oct 30 01:24:54.243008 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.242988 4927 scope.go:122] "RemoveContainer" containerID="3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426" Oct 30 01:24:54.243185 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:24:54.243170 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426\": container with ID starting with 3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426 not found: ID does not exist" containerID="3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426" Oct 30 01:24:54.243227 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.243189 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426"} err="failed to get container status \"3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426\": rpc error: code = NotFound desc = could not find container \"3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426\": container with ID starting with 3f9d57bc25e4bd1109d570c9ddb658bc427b23145a6ae613bb9e3ea1c2971426 not found: ID does not exist" Oct 30 01:24:54.283518 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.283487 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7"] Oct 30 01:24:54.294923 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:54.294898 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e9921-openshift-default-7d98cc65c5-xr8j7"] Oct 30 01:24:55.571854 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:55.571825 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" path="/var/lib/kubelet/pods/4836cefc-d5ca-469d-899b-d9298b44db87/volumes" Oct 30 01:24:58.866863 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:58.866831 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c"] Oct 30 01:24:58.867350 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:58.867040 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" containerID="cri-o://d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1" gracePeriod=30 Oct 30 01:24:59.200058 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:59.200018 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v"] Oct 30 01:24:59.235585 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:24:59.235532 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w"] Oct 30 01:25:03.463249 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.463212 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp"] Oct 30 01:25:03.463811 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.463505 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" podUID="f3fbc92f-3abd-4859-a96c-70d5c6e7078c" containerName="maas-api" containerID="cri-o://a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd" gracePeriod=30 Oct 30 01:25:03.522525 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.522479 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" podUID="f3fbc92f-3abd-4859-a96c-70d5c6e7078c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.127:8443/health\": dial tcp 10.132.0.127:8443: connect: connection refused" Oct 30 01:25:03.745203 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.745170 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 01:25:03.824404 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.824366 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx"] Oct 30 01:25:03.824660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.824625 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" podUID="80835022-7789-49ed-b568-c8f28ae3ae2c" containerName="maas-api" containerID="cri-o://fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7" gracePeriod=30 Oct 30 01:25:03.844858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.844004 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls\") pod \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " Oct 30 01:25:03.844858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.844072 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-kube-api-access-f27fb\" (UniqueName: \"kubernetes.io/projected/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-kube-api-access-f27fb\") pod \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " Oct 30 01:25:03.844858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.844119 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-openshift-service-ca\") pod \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " Oct 30 01:25:03.844858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.844230 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-metrics-certs\") pod \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\" (UID: \"f3fbc92f-3abd-4859-a96c-70d5c6e7078c\") " Oct 30 01:25:03.845924 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.845888 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-openshift-service-ca" pod "f3fbc92f-3abd-4859-a96c-70d5c6e7078c" (UID: "f3fbc92f-3abd-4859-a96c-70d5c6e7078c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:03.847487 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.847449 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-metrics-certs" pod "f3fbc92f-3abd-4859-a96c-70d5c6e7078c" (UID: "f3fbc92f-3abd-4859-a96c-70d5c6e7078c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:03.848396 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.848359 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls" pod "f3fbc92f-3abd-4859-a96c-70d5c6e7078c" (UID: "f3fbc92f-3abd-4859-a96c-70d5c6e7078c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:03.850386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.850351 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-kube-api-access-f27fb" pod "f3fbc92f-3abd-4859-a96c-70d5c6e7078c" (UID: "f3fbc92f-3abd-4859-a96c-70d5c6e7078c"). InnerVolumeSpecName "kube-api-access-f27fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:03.945633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.945594 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:03.945633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.945631 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:03.945831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.945641 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f27fb\" (UniqueName: \"kubernetes.io/projected/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-kube-api-access-f27fb\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:03.945831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.945667 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fbc92f-3abd-4859-a96c-70d5c6e7078c-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:03.990507 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.990473 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7"] Oct 30 01:25:03.990734 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:03.990710 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" podUID="23d22d03-5652-4aa6-a40c-6de31a390844" containerName="payload-processing" containerID="cri-o://31e1b784f841b199715c38db71fe0f3284a7a658e04aac18bed842f0c65c1492" gracePeriod=30 Oct 30 01:25:04.047635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.036086 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m"] Oct 30 01:25:04.047635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.036331 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" podUID="b3987a5c-3a1d-4602-af47-7fb88849fa9e" containerName="payload-pre-processing" containerID="cri-o://8940af674b3eafb1f2a521107c43951199fcec1ab6a1a2c6b676d533b1e1a2bb" gracePeriod=30 Oct 30 01:25:04.193284 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.192947 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:25:04.205112 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.202269 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" podUID="ff872648-3c7b-4b11-90b1-8d34939b17ce" containerName="payload-processing" containerID="cri-o://a2e19378d9e052fb44629c34170b01047012a84f959b6ba676d291ed2cc0af39" gracePeriod=25 Oct 30 01:25:04.223633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.223566 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 01:25:04.237389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.236842 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" podUID="651c823f-662b-404a-9374-c0f641f322df" containerName="payload-pre-processing" containerID="cri-o://f1feecee5b81e8bdf6a7c5577a64235bffd47eb685ee2afc68fddf36812cc655" gracePeriod=25 Oct 30 01:25:04.293021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.292496 4927 generic.go:356] "Generic (PLEG): container finished" podID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerID="d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1" exitCode=0 Oct 30 01:25:04.293021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.292694 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" event={"ID":"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde","Type":"ContainerDied","Data":"d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1"} Oct 30 01:25:04.293021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.292753 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" event={"ID":"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde","Type":"ContainerDied","Data":"c1d3eda0ea57c6ff11d0f7a7fd58e974461fddff96658a686689180e5635f9dd"} Oct 30 01:25:04.293021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.292778 4927 scope.go:122] "RemoveContainer" containerID="d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1" Oct 30 01:25:04.293021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.292912 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c" Oct 30 01:25:04.295606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.295545 4927 generic.go:356] "Generic (PLEG): container finished" podID="b3987a5c-3a1d-4602-af47-7fb88849fa9e" containerID="8940af674b3eafb1f2a521107c43951199fcec1ab6a1a2c6b676d533b1e1a2bb" exitCode=0 Oct 30 01:25:04.295786 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.295627 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" event={"ID":"b3987a5c-3a1d-4602-af47-7fb88849fa9e","Type":"ContainerDied","Data":"8940af674b3eafb1f2a521107c43951199fcec1ab6a1a2c6b676d533b1e1a2bb"} Oct 30 01:25:04.297345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.297150 4927 generic.go:356] "Generic (PLEG): container finished" podID="80835022-7789-49ed-b568-c8f28ae3ae2c" containerID="fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7" exitCode=0 Oct 30 01:25:04.297345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.297205 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" Oct 30 01:25:04.297345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.297235 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" event={"ID":"80835022-7789-49ed-b568-c8f28ae3ae2c","Type":"ContainerDied","Data":"fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7"} Oct 30 01:25:04.297345 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.297291 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx" event={"ID":"80835022-7789-49ed-b568-c8f28ae3ae2c","Type":"ContainerDied","Data":"b89d2b41397bb396b1a237e09b603aabc9190dea5df6a6b5c655b8a631a181bf"} Oct 30 01:25:04.299361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.299026 4927 generic.go:356] "Generic (PLEG): container finished" podID="23d22d03-5652-4aa6-a40c-6de31a390844" containerID="31e1b784f841b199715c38db71fe0f3284a7a658e04aac18bed842f0c65c1492" exitCode=0 Oct 30 01:25:04.299361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.299095 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" event={"ID":"23d22d03-5652-4aa6-a40c-6de31a390844","Type":"ContainerDied","Data":"31e1b784f841b199715c38db71fe0f3284a7a658e04aac18bed842f0c65c1492"} Oct 30 01:25:04.302598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.300782 4927 generic.go:356] "Generic (PLEG): container finished" podID="f3fbc92f-3abd-4859-a96c-70d5c6e7078c" containerID="a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd" exitCode=0 Oct 30 01:25:04.302598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.300880 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" event={"ID":"f3fbc92f-3abd-4859-a96c-70d5c6e7078c","Type":"ContainerDied","Data":"a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd"} Oct 30 01:25:04.302598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.300916 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" event={"ID":"f3fbc92f-3abd-4859-a96c-70d5c6e7078c","Type":"ContainerDied","Data":"25c55d9bbbb6bfd68e966e5e740c0bff3068bc0c0bff3d3dfdbd4f4a35283110"} Oct 30 01:25:04.302598 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.300977 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp" Oct 30 01:25:04.309774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.309751 4927 scope.go:122] "RemoveContainer" containerID="d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1" Oct 30 01:25:04.310293 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:04.310139 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1\": container with ID starting with d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1 not found: ID does not exist" containerID="d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1" Oct 30 01:25:04.310293 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.310175 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1"} err="failed to get container status \"d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1\": rpc error: code = NotFound desc = could not find container \"d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1\": container with ID starting with d5b0193a304b9fe454a8573ced76e8b976e8b294301ac02e11686e5eed9bd8a1 not found: ID does not exist" Oct 30 01:25:04.310293 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.310200 4927 scope.go:122] "RemoveContainer" containerID="fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7" Oct 30 01:25:04.326428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.326362 4927 scope.go:122] "RemoveContainer" containerID="fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7" Oct 30 01:25:04.326773 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:04.326744 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7\": container with ID starting with fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7 not found: ID does not exist" containerID="fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7" Oct 30 01:25:04.326875 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.326783 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7"} err="failed to get container status \"fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7\": rpc error: code = NotFound desc = could not find container \"fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7\": container with ID starting with fc9e645fe33d157e1708eac15f584392e1925392231461bb5af5165b600cecd7 not found: ID does not exist" Oct 30 01:25:04.326875 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.326806 4927 scope.go:122] "RemoveContainer" containerID="a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd" Oct 30 01:25:04.338208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.338178 4927 scope.go:122] "RemoveContainer" containerID="a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd" Oct 30 01:25:04.338511 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:04.338480 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd\": container with ID starting with a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd not found: ID does not exist" containerID="a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd" Oct 30 01:25:04.338596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.338520 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd"} err="failed to get container status \"a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd\": rpc error: code = NotFound desc = could not find container \"a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd\": container with ID starting with a6487b33cb78c93db005923fefb643462b0c81e3758632224cbc9b11dff8cbfd not found: ID does not exist" Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.356859 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-envoy\") pod \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.356950 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-metrics-certs\") pod \"80835022-7789-49ed-b568-c8f28ae3ae2c\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357012 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-podinfo\") pod \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357034 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-socket\") pod \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357076 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80835022-7789-49ed-b568-c8f28ae3ae2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80835022-7789-49ed-b568-c8f28ae3ae2c-openshift-service-ca\") pod \"80835022-7789-49ed-b568-c8f28ae3ae2c\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357122 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-kube-api-access-s99lt\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-kube-api-access-s99lt\") pod \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357147 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-token\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-token\") pod \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357169 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80835022-7789-49ed-b568-c8f28ae3ae2c-kube-api-access-cks46\" (UniqueName: \"kubernetes.io/projected/80835022-7789-49ed-b568-c8f28ae3ae2c-kube-api-access-cks46\") pod \"80835022-7789-49ed-b568-c8f28ae3ae2c\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357188 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istiod-ca-cert\") pod \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357208 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-credential-socket\") pod \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357237 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-certs\") pod \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357263 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-data\") pod \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\" (UID: \"7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde\") " Oct 30 01:25:04.357990 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.357289 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-maas-api-tls\") pod \"80835022-7789-49ed-b568-c8f28ae3ae2c\" (UID: \"80835022-7789-49ed-b568-c8f28ae3ae2c\") " Oct 30 01:25:04.359139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.358134 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-certs" pod "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" (UID: "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.359139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.358364 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-data" pod "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" (UID: "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.359139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.358714 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-socket" pod "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" (UID: "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.359139 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.358848 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istiod-ca-cert" pod "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" (UID: "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:04.359374 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.359121 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-credential-socket" pod "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" (UID: "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.359374 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.359258 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80835022-7789-49ed-b568-c8f28ae3ae2c-openshift-service-ca" pod "80835022-7789-49ed-b568-c8f28ae3ae2c" (UID: "80835022-7789-49ed-b568-c8f28ae3ae2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:04.360182 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.360153 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-token" pod "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" (UID: "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:04.361728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.361694 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-envoy" pod "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" (UID: "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:04.361999 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.361965 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-maas-api-tls" pod "80835022-7789-49ed-b568-c8f28ae3ae2c" (UID: "80835022-7789-49ed-b568-c8f28ae3ae2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:04.362212 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.362187 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-metrics-certs" pod "80835022-7789-49ed-b568-c8f28ae3ae2c" (UID: "80835022-7789-49ed-b568-c8f28ae3ae2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:04.362357 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.362324 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80835022-7789-49ed-b568-c8f28ae3ae2c-kube-api-access-cks46" pod "80835022-7789-49ed-b568-c8f28ae3ae2c" (UID: "80835022-7789-49ed-b568-c8f28ae3ae2c"). InnerVolumeSpecName "kube-api-access-cks46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:04.362956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.362933 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-podinfo" pod "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" (UID: "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:04.363261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.363240 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-kube-api-access-s99lt" pod "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" (UID: "7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde"). InnerVolumeSpecName "kube-api-access-s99lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:04.382361 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.382323 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:25:04.432952 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.432696 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp"] Oct 30 01:25:04.433982 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.433957 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-95492e-6fbf4dbfb4-2l4gp"] Oct 30 01:25:04.442187 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.442148 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458365 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3987a5c-3a1d-4602-af47-7fb88849fa9e-kube-api-access-6dwcl\" (UniqueName: \"kubernetes.io/projected/b3987a5c-3a1d-4602-af47-7fb88849fa9e-kube-api-access-6dwcl\") pod \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\" (UID: \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\") " Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458412 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d22d03-5652-4aa6-a40c-6de31a390844-kube-api-access-n699m\" (UniqueName: \"kubernetes.io/projected/23d22d03-5652-4aa6-a40c-6de31a390844-kube-api-access-n699m\") pod \"23d22d03-5652-4aa6-a40c-6de31a390844\" (UID: \"23d22d03-5652-4aa6-a40c-6de31a390844\") " Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458452 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3987a5c-3a1d-4602-af47-7fb88849fa9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3987a5c-3a1d-4602-af47-7fb88849fa9e-plugins-config-volume\") pod \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\" (UID: \"b3987a5c-3a1d-4602-af47-7fb88849fa9e\") " Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458500 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d22d03-5652-4aa6-a40c-6de31a390844-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d22d03-5652-4aa6-a40c-6de31a390844-plugins-config-volume\") pod \"23d22d03-5652-4aa6-a40c-6de31a390844\" (UID: \"23d22d03-5652-4aa6-a40c-6de31a390844\") " Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458803 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99lt\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-kube-api-access-s99lt\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458818 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458827 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cks46\" (UniqueName: \"kubernetes.io/projected/80835022-7789-49ed-b568-c8f28ae3ae2c-kube-api-access-cks46\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458835 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458845 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458853 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458862 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458869 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458878 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458885 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80835022-7789-49ed-b568-c8f28ae3ae2c-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458892 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458901 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.459095 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.458908 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80835022-7789-49ed-b568-c8f28ae3ae2c-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.461375 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.460719 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d22d03-5652-4aa6-a40c-6de31a390844-kube-api-access-n699m" pod "23d22d03-5652-4aa6-a40c-6de31a390844" (UID: "23d22d03-5652-4aa6-a40c-6de31a390844"). InnerVolumeSpecName "kube-api-access-n699m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:04.461375 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.461175 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d22d03-5652-4aa6-a40c-6de31a390844-plugins-config-volume" pod "23d22d03-5652-4aa6-a40c-6de31a390844" (UID: "23d22d03-5652-4aa6-a40c-6de31a390844"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:04.463137 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.463110 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3987a5c-3a1d-4602-af47-7fb88849fa9e-plugins-config-volume" pod "b3987a5c-3a1d-4602-af47-7fb88849fa9e" (UID: "b3987a5c-3a1d-4602-af47-7fb88849fa9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:04.464690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.464667 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3987a5c-3a1d-4602-af47-7fb88849fa9e-kube-api-access-6dwcl" pod "b3987a5c-3a1d-4602-af47-7fb88849fa9e" (UID: "b3987a5c-3a1d-4602-af47-7fb88849fa9e"). InnerVolumeSpecName "kube-api-access-6dwcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:04.470481 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:04.470453 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:25:04.560036 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.560004 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d22d03-5652-4aa6-a40c-6de31a390844-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.560036 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.560033 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwcl\" (UniqueName: \"kubernetes.io/projected/b3987a5c-3a1d-4602-af47-7fb88849fa9e-kube-api-access-6dwcl\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.560243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.560043 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n699m\" (UniqueName: \"kubernetes.io/projected/23d22d03-5652-4aa6-a40c-6de31a390844-kube-api-access-n699m\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.560243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.560055 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3987a5c-3a1d-4602-af47-7fb88849fa9e-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.640447 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.640413 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c"] Oct 30 01:25:04.644538 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.644510 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jj77c"] Oct 30 01:25:04.680775 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.680743 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx"] Oct 30 01:25:04.684378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:04.684344 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc3cb2-5d79ffc959-57bdx"] Oct 30 01:25:05.310067 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.310031 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" event={"ID":"b3987a5c-3a1d-4602-af47-7fb88849fa9e","Type":"ContainerDied","Data":"08bd5d8b52146bc27439058d6944ae2a0406e0f2aeebe9aba34e9197cd9fecfc"} Oct 30 01:25:05.310244 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.310051 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m" Oct 30 01:25:05.310244 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.310142 4927 scope.go:122] "RemoveContainer" containerID="8940af674b3eafb1f2a521107c43951199fcec1ab6a1a2c6b676d533b1e1a2bb" Oct 30 01:25:05.312828 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.312805 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" Oct 30 01:25:05.312828 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.312819 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7" event={"ID":"23d22d03-5652-4aa6-a40c-6de31a390844","Type":"ContainerDied","Data":"fb326a879cc4ee8c535680a849e0a0324422432b6928907711fadd2bfd0bda41"} Oct 30 01:25:05.322233 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.322216 4927 scope.go:122] "RemoveContainer" containerID="31e1b784f841b199715c38db71fe0f3284a7a658e04aac18bed842f0c65c1492" Oct 30 01:25:05.367138 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.367108 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7"] Oct 30 01:25:05.371311 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.371279 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc3cb2-58bc75d6-cppx7"] Oct 30 01:25:05.408488 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.408454 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m"] Oct 30 01:25:05.412315 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.412292 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc3cb2-645847cb7f-wz49m"] Oct 30 01:25:05.571089 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.570997 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d22d03-5652-4aa6-a40c-6de31a390844" path="/var/lib/kubelet/pods/23d22d03-5652-4aa6-a40c-6de31a390844/volumes" Oct 30 01:25:05.571592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.571561 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" path="/var/lib/kubelet/pods/7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde/volumes" Oct 30 01:25:05.571975 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.571962 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80835022-7789-49ed-b568-c8f28ae3ae2c" path="/var/lib/kubelet/pods/80835022-7789-49ed-b568-c8f28ae3ae2c/volumes" Oct 30 01:25:05.572665 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.572649 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3987a5c-3a1d-4602-af47-7fb88849fa9e" path="/var/lib/kubelet/pods/b3987a5c-3a1d-4602-af47-7fb88849fa9e/volumes" Oct 30 01:25:05.573046 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:05.573033 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3fbc92f-3abd-4859-a96c-70d5c6e7078c" path="/var/lib/kubelet/pods/f3fbc92f-3abd-4859-a96c-70d5c6e7078c/volumes" Oct 30 01:25:08.900815 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:08.900777 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" podUID="651c823f-662b-404a-9374-c0f641f322df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:08.901217 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:08.900777 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" podUID="ff872648-3c7b-4b11-90b1-8d34939b17ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:11.911938 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.911899 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd"] Oct 30 01:25:11.912445 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912425 4927 cpu_manager.go:414] "containerMap: removing container" podUID="5a52d418-6bef-48df-841e-5b55dd4e42ac" containerName="maas-api" Oct 30 01:25:11.912445 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912441 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a52d418-6bef-48df-841e-5b55dd4e42ac" containerName="maas-api" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912462 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b04c018e-d268-4c73-97c2-63436cebfd55" containerName="maas-api" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912468 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04c018e-d268-4c73-97c2-63436cebfd55" containerName="maas-api" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912479 4927 cpu_manager.go:414] "containerMap: removing container" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912485 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912500 4927 cpu_manager.go:414] "containerMap: removing container" podUID="888a0308-4626-478a-80c7-efcdcb04cf19" containerName="main" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912505 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888a0308-4626-478a-80c7-efcdcb04cf19" containerName="main" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912513 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1cd85074-3f4b-4d36-ad24-fa7abc24a367" containerName="payload-pre-processing" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912517 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd85074-3f4b-4d36-ad24-fa7abc24a367" containerName="payload-pre-processing" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912524 4927 cpu_manager.go:414] "containerMap: removing container" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912528 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912536 4927 cpu_manager.go:414] "containerMap: removing container" podUID="97acea83-49aa-4c90-bf1c-312dc1dd7d02" containerName="payload-processing" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912541 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97acea83-49aa-4c90-bf1c-312dc1dd7d02" containerName="payload-processing" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912555 4927 cpu_manager.go:414] "containerMap: removing container" podUID="65ce4978-b441-48c9-9145-24c6bdf788f6" containerName="payload-processing" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912560 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ce4978-b441-48c9-9145-24c6bdf788f6" containerName="payload-processing" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912581 4927 cpu_manager.go:414] "containerMap: removing container" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912587 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912596 4927 cpu_manager.go:414] "containerMap: removing container" podUID="febc6e30-896a-495a-af9c-7a6a65e40975" containerName="maas-api" Oct 30 01:25:11.912588 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912600 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febc6e30-896a-495a-af9c-7a6a65e40975" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912606 4927 cpu_manager.go:414] "containerMap: removing container" podUID="23d22d03-5652-4aa6-a40c-6de31a390844" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912610 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d22d03-5652-4aa6-a40c-6de31a390844" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912620 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a8d33d24-b5e3-4e1d-b039-b14a8ca03432" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912625 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d33d24-b5e3-4e1d-b039-b14a8ca03432" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912631 4927 cpu_manager.go:414] "containerMap: removing container" podUID="80835022-7789-49ed-b568-c8f28ae3ae2c" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912635 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80835022-7789-49ed-b568-c8f28ae3ae2c" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912644 4927 cpu_manager.go:414] "containerMap: removing container" podUID="dbaf19a5-5ea9-4353-80e8-333668ef5bde" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912648 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbaf19a5-5ea9-4353-80e8-333668ef5bde" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912655 4927 cpu_manager.go:414] "containerMap: removing container" podUID="29ab4ce5-416f-4f47-832a-da72d1263042" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912659 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ab4ce5-416f-4f47-832a-da72d1263042" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912666 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912670 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912679 4927 cpu_manager.go:414] "containerMap: removing container" podUID="269e1e40-2de3-4384-8d31-282bfb066435" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912683 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269e1e40-2de3-4384-8d31-282bfb066435" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912690 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f3fbc92f-3abd-4859-a96c-70d5c6e7078c" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912694 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3fbc92f-3abd-4859-a96c-70d5c6e7078c" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912701 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912706 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912712 4927 cpu_manager.go:414] "containerMap: removing container" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912716 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912725 4927 cpu_manager.go:414] "containerMap: removing container" podUID="b3987a5c-3a1d-4602-af47-7fb88849fa9e" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912731 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3987a5c-3a1d-4602-af47-7fb88849fa9e" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912737 4927 cpu_manager.go:414] "containerMap: removing container" podUID="414818dd-4954-4548-be30-01f1a9ded7c5" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912742 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414818dd-4954-4548-be30-01f1a9ded7c5" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912749 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ba09d4f5-5e04-4733-8e95-65527ab7ca15" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912753 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba09d4f5-5e04-4733-8e95-65527ab7ca15" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912759 4927 cpu_manager.go:414] "containerMap: removing container" podUID="82cb426a-d010-427a-a153-569fee89d6a7" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912764 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82cb426a-d010-427a-a153-569fee89d6a7" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912770 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912775 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912785 4927 cpu_manager.go:414] "containerMap: removing container" podUID="cd70c464-1507-414d-96d9-a138e4143281" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912789 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd70c464-1507-414d-96d9-a138e4143281" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912798 4927 cpu_manager.go:414] "containerMap: removing container" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912802 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912888 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="269e1e40-2de3-4384-8d31-282bfb066435" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912896 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3fbc92f-3abd-4859-a96c-70d5c6e7078c" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912902 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912909 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7e7ec9-3153-438f-820b-c3e8e35bd040" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912914 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3987a5c-3a1d-4602-af47-7fb88849fa9e" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912921 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="414818dd-4954-4548-be30-01f1a9ded7c5" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912928 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba09d4f5-5e04-4733-8e95-65527ab7ca15" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912933 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="82cb426a-d010-427a-a153-569fee89d6a7" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912938 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94498ce-f9f6-4586-8324-e1c5cd85c4f4" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912944 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd70c464-1507-414d-96d9-a138e4143281" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912950 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="4836cefc-d5ca-469d-899b-d9298b44db87" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912955 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a52d418-6bef-48df-841e-5b55dd4e42ac" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912961 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04c018e-d268-4c73-97c2-63436cebfd55" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912966 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912973 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="888a0308-4626-478a-80c7-efcdcb04cf19" containerName="main" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912979 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd85074-3f4b-4d36-ad24-fa7abc24a367" containerName="payload-pre-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912984 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e21533-1c44-41fa-82a3-f5ed8296bdb4" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912990 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="97acea83-49aa-4c90-bf1c-312dc1dd7d02" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.912997 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ce4978-b441-48c9-9145-24c6bdf788f6" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.913003 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da6e5d2-5ca6-4c6a-9ae6-8c12457f3bde" containerName="istio-proxy" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.913008 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="febc6e30-896a-495a-af9c-7a6a65e40975" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.913013 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d22d03-5652-4aa6-a40c-6de31a390844" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.913020 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d33d24-b5e3-4e1d-b039-b14a8ca03432" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.913024 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="80835022-7789-49ed-b568-c8f28ae3ae2c" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.913030 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbaf19a5-5ea9-4353-80e8-333668ef5bde" containerName="payload-processing" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.913035 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ab4ce5-416f-4f47-832a-da72d1263042" containerName="maas-api" Oct 30 01:25:11.913609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.913041 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffb6932-005f-4829-a8c0-9ff8801be789" containerName="istio-proxy" Oct 30 01:25:11.918027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.918009 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:11.938784 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:11.938758 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd"] Oct 30 01:25:12.013296 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.013259 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:25:12.013509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.013489 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:25:12.042254 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.042222 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.042415 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.042256 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.042505 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.042481 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd4v4\" (UniqueName: \"kubernetes.io/projected/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-kube-api-access-sd4v4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.143814 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.143773 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.144020 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.143820 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.144094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.144016 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd4v4\" (UniqueName: \"kubernetes.io/projected/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-kube-api-access-sd4v4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.144642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.144613 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.146232 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.146213 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.166440 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.166357 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd4v4\" (UniqueName: \"kubernetes.io/projected/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-kube-api-access-sd4v4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.231735 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.231703 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:12.426457 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.426380 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d"] Oct 30 01:25:12.426649 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.426619 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" containerID="cri-o://6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb" gracePeriod=30 Oct 30 01:25:12.542136 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:12.542093 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd"] Oct 30 01:25:12.553496 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:25:12.553449 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51d8b20_a2bc_47d5_befa_55eb443e7ba2.slice/crio-6fce23a11b5d2e06d85c1d89d1f97e4cf248bf602b05b196b805f9e64d9e2107 WatchSource:0}: Error finding container 6fce23a11b5d2e06d85c1d89d1f97e4cf248bf602b05b196b805f9e64d9e2107: Status 404 returned error can't find the container with id 6fce23a11b5d2e06d85c1d89d1f97e4cf248bf602b05b196b805f9e64d9e2107 Oct 30 01:25:13.372418 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:13.372363 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" event={"ID":"e51d8b20-a2bc-47d5-befa-55eb443e7ba2","Type":"ContainerStarted","Data":"e774eb7b5cc1de5a104ccc5a3bb6812e7a38507fbeeddde975a15f7f809f72ef"} Oct 30 01:25:13.372877 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:13.372436 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" event={"ID":"e51d8b20-a2bc-47d5-befa-55eb443e7ba2","Type":"ContainerStarted","Data":"6fce23a11b5d2e06d85c1d89d1f97e4cf248bf602b05b196b805f9e64d9e2107"} Oct 30 01:25:13.411987 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:13.411925 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" podStartSLOduration=2.411906679 podStartE2EDuration="2.411906679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:13.411043519 +0000 UTC m=+3051.982969754" watchObservedRunningTime="2026-10-30 01:25:13.411906679 +0000 UTC m=+3051.983832902" Oct 30 01:25:14.380399 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:14.380365 4927 generic.go:356] "Generic (PLEG): container finished" podID="e51d8b20-a2bc-47d5-befa-55eb443e7ba2" containerID="e774eb7b5cc1de5a104ccc5a3bb6812e7a38507fbeeddde975a15f7f809f72ef" exitCode=0 Oct 30 01:25:14.380793 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:14.380457 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" event={"ID":"e51d8b20-a2bc-47d5-befa-55eb443e7ba2","Type":"ContainerDied","Data":"e774eb7b5cc1de5a104ccc5a3bb6812e7a38507fbeeddde975a15f7f809f72ef"} Oct 30 01:25:15.521489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.521459 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:15.585636 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.585598 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-openshift-ca\") pod \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " Oct 30 01:25:15.585839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.585723 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-xks-ca\") pod \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " Oct 30 01:25:15.585839 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.585802 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-kube-api-access-sd4v4\" (UniqueName: \"kubernetes.io/projected/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-kube-api-access-sd4v4\") pod \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\" (UID: \"e51d8b20-a2bc-47d5-befa-55eb443e7ba2\") " Oct 30 01:25:15.585973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.585908 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-openshift-ca" pod "e51d8b20-a2bc-47d5-befa-55eb443e7ba2" (UID: "e51d8b20-a2bc-47d5-befa-55eb443e7ba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:15.586264 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.586240 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-openshift-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:15.587994 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.587960 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-xks-ca" pod "e51d8b20-a2bc-47d5-befa-55eb443e7ba2" (UID: "e51d8b20-a2bc-47d5-befa-55eb443e7ba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:15.588092 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.588036 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-kube-api-access-sd4v4" pod "e51d8b20-a2bc-47d5-befa-55eb443e7ba2" (UID: "e51d8b20-a2bc-47d5-befa-55eb443e7ba2"). InnerVolumeSpecName "kube-api-access-sd4v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:15.687243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.687159 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-maas-api-xks-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:15.687243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:15.687194 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd4v4\" (UniqueName: \"kubernetes.io/projected/e51d8b20-a2bc-47d5-befa-55eb443e7ba2-kube-api-access-sd4v4\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:16.396330 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:16.396299 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" Oct 30 01:25:16.396523 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:16.396309 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd" event={"ID":"e51d8b20-a2bc-47d5-befa-55eb443e7ba2","Type":"ContainerDied","Data":"6fce23a11b5d2e06d85c1d89d1f97e4cf248bf602b05b196b805f9e64d9e2107"} Oct 30 01:25:16.396523 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:16.396418 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fce23a11b5d2e06d85c1d89d1f97e4cf248bf602b05b196b805f9e64d9e2107" Oct 30 01:25:17.695332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.695250 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:25:17.809661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.809625 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-socket\") pod \"1038975e-5fd3-4674-9374-e4a46799b7ff\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " Oct 30 01:25:17.809858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.809689 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-certs\") pod \"1038975e-5fd3-4674-9374-e4a46799b7ff\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " Oct 30 01:25:17.809858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.809751 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1038975e-5fd3-4674-9374-e4a46799b7ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1038975e-5fd3-4674-9374-e4a46799b7ff-istiod-ca-cert\") pod \"1038975e-5fd3-4674-9374-e4a46799b7ff\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " Oct 30 01:25:17.809858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.809784 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-envoy\") pod \"1038975e-5fd3-4674-9374-e4a46799b7ff\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " Oct 30 01:25:17.809858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.809838 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-data\") pod \"1038975e-5fd3-4674-9374-e4a46799b7ff\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " Oct 30 01:25:17.810117 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.809903 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-credential-socket\") pod \"1038975e-5fd3-4674-9374-e4a46799b7ff\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " Oct 30 01:25:17.810117 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.809938 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-podinfo\") pod \"1038975e-5fd3-4674-9374-e4a46799b7ff\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " Oct 30 01:25:17.810117 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.809943 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-socket" pod "1038975e-5fd3-4674-9374-e4a46799b7ff" (UID: "1038975e-5fd3-4674-9374-e4a46799b7ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:17.810117 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.809976 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-token\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-token\") pod \"1038975e-5fd3-4674-9374-e4a46799b7ff\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " Oct 30 01:25:17.810117 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810033 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-kube-api-access-xlzpf\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-kube-api-access-xlzpf\") pod \"1038975e-5fd3-4674-9374-e4a46799b7ff\" (UID: \"1038975e-5fd3-4674-9374-e4a46799b7ff\") " Oct 30 01:25:17.810117 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810078 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-certs" pod "1038975e-5fd3-4674-9374-e4a46799b7ff" (UID: "1038975e-5fd3-4674-9374-e4a46799b7ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:17.810512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810207 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-credential-socket" pod "1038975e-5fd3-4674-9374-e4a46799b7ff" (UID: "1038975e-5fd3-4674-9374-e4a46799b7ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:17.810512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810253 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1038975e-5fd3-4674-9374-e4a46799b7ff-istiod-ca-cert" pod "1038975e-5fd3-4674-9374-e4a46799b7ff" (UID: "1038975e-5fd3-4674-9374-e4a46799b7ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:17.810512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810396 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-data" pod "1038975e-5fd3-4674-9374-e4a46799b7ff" (UID: "1038975e-5fd3-4674-9374-e4a46799b7ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:17.810512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810490 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:17.810512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810511 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:17.810800 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810525 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1038975e-5fd3-4674-9374-e4a46799b7ff-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:17.810800 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810537 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:17.810800 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.810548 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:17.812324 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.812301 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-envoy" pod "1038975e-5fd3-4674-9374-e4a46799b7ff" (UID: "1038975e-5fd3-4674-9374-e4a46799b7ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:17.812420 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.812379 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-token" pod "1038975e-5fd3-4674-9374-e4a46799b7ff" (UID: "1038975e-5fd3-4674-9374-e4a46799b7ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:17.812618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.812596 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-podinfo" pod "1038975e-5fd3-4674-9374-e4a46799b7ff" (UID: "1038975e-5fd3-4674-9374-e4a46799b7ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:17.812618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.812607 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-kube-api-access-xlzpf" pod "1038975e-5fd3-4674-9374-e4a46799b7ff" (UID: "1038975e-5fd3-4674-9374-e4a46799b7ff"). InnerVolumeSpecName "kube-api-access-xlzpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:17.912031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.911996 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:17.912031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.912025 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:17.912031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.912036 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzpf\" (UniqueName: \"kubernetes.io/projected/1038975e-5fd3-4674-9374-e4a46799b7ff-kube-api-access-xlzpf\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:17.912279 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:17.912050 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1038975e-5fd3-4674-9374-e4a46799b7ff-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:18.410777 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.410741 4927 generic.go:356] "Generic (PLEG): container finished" podID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerID="6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb" exitCode=0 Oct 30 01:25:18.410968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.410793 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" Oct 30 01:25:18.410968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.410824 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" event={"ID":"1038975e-5fd3-4674-9374-e4a46799b7ff","Type":"ContainerDied","Data":"6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb"} Oct 30 01:25:18.410968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.410880 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d" event={"ID":"1038975e-5fd3-4674-9374-e4a46799b7ff","Type":"ContainerDied","Data":"71e326bedab2894d26d53d84806d2fc0cc40056874be0b7fe9c63ba84cdd0d52"} Oct 30 01:25:18.410968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.410902 4927 scope.go:122] "RemoveContainer" containerID="6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb" Oct 30 01:25:18.421803 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.421786 4927 scope.go:122] "RemoveContainer" containerID="35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02" Oct 30 01:25:18.431431 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.431410 4927 scope.go:122] "RemoveContainer" containerID="6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb" Oct 30 01:25:18.431709 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:18.431689 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb\": container with ID starting with 6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb not found: ID does not exist" containerID="6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb" Oct 30 01:25:18.431802 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.431723 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb"} err="failed to get container status \"6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb\": rpc error: code = NotFound desc = could not find container \"6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb\": container with ID starting with 6687fcf66a7b66ae0b41caf567305934532cccb5dd44bdeab9f15ae594731dbb not found: ID does not exist" Oct 30 01:25:18.431802 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.431748 4927 scope.go:122] "RemoveContainer" containerID="35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02" Oct 30 01:25:18.431997 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:18.431978 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02\": container with ID starting with 35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02 not found: ID does not exist" containerID="35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02" Oct 30 01:25:18.432038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.432005 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02"} err="failed to get container status \"35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02\": rpc error: code = NotFound desc = could not find container \"35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02\": container with ID starting with 35a9953d2383ffc179e13d60d98f7789129500d451572fc10cdb8fbbb2ea8c02 not found: ID does not exist" Oct 30 01:25:18.472964 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.472935 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d"] Oct 30 01:25:18.484036 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.484013 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc3cb2-openshift-default-69bbd5978c-jx55d"] Oct 30 01:25:18.901394 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.901344 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" podUID="ff872648-3c7b-4b11-90b1-8d34939b17ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:18.901842 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:18.901357 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" podUID="651c823f-662b-404a-9374-c0f641f322df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:19.570850 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:19.570817 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" path="/var/lib/kubelet/pods/1038975e-5fd3-4674-9374-e4a46799b7ff/volumes" Oct 30 01:25:23.426075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:23.426030 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b"] Oct 30 01:25:24.454011 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.453971 4927 generic.go:356] "Generic (PLEG): container finished" podID="651c823f-662b-404a-9374-c0f641f322df" containerID="f1feecee5b81e8bdf6a7c5577a64235bffd47eb685ee2afc68fddf36812cc655" exitCode=0 Oct 30 01:25:24.454441 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.454040 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" event={"ID":"651c823f-662b-404a-9374-c0f641f322df","Type":"ContainerDied","Data":"f1feecee5b81e8bdf6a7c5577a64235bffd47eb685ee2afc68fddf36812cc655"} Oct 30 01:25:24.456116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.456082 4927 generic.go:356] "Generic (PLEG): container finished" podID="ff872648-3c7b-4b11-90b1-8d34939b17ce" containerID="a2e19378d9e052fb44629c34170b01047012a84f959b6ba676d291ed2cc0af39" exitCode=0 Oct 30 01:25:24.456351 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.456330 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" event={"ID":"ff872648-3c7b-4b11-90b1-8d34939b17ce","Type":"ContainerDied","Data":"a2e19378d9e052fb44629c34170b01047012a84f959b6ba676d291ed2cc0af39"} Oct 30 01:25:24.503998 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.503972 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 01:25:24.530767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.530743 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 01:25:24.583944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.583913 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff872648-3c7b-4b11-90b1-8d34939b17ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff872648-3c7b-4b11-90b1-8d34939b17ce-plugins-config-volume\") pod \"ff872648-3c7b-4b11-90b1-8d34939b17ce\" (UID: \"ff872648-3c7b-4b11-90b1-8d34939b17ce\") " Oct 30 01:25:24.583944 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.583949 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff872648-3c7b-4b11-90b1-8d34939b17ce-kube-api-access-tn4bc\" (UniqueName: \"kubernetes.io/projected/ff872648-3c7b-4b11-90b1-8d34939b17ce-kube-api-access-tn4bc\") pod \"ff872648-3c7b-4b11-90b1-8d34939b17ce\" (UID: \"ff872648-3c7b-4b11-90b1-8d34939b17ce\") " Oct 30 01:25:24.584384 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.584351 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff872648-3c7b-4b11-90b1-8d34939b17ce-plugins-config-volume" pod "ff872648-3c7b-4b11-90b1-8d34939b17ce" (UID: "ff872648-3c7b-4b11-90b1-8d34939b17ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:24.587659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.587635 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff872648-3c7b-4b11-90b1-8d34939b17ce-kube-api-access-tn4bc" pod "ff872648-3c7b-4b11-90b1-8d34939b17ce" (UID: "ff872648-3c7b-4b11-90b1-8d34939b17ce"). InnerVolumeSpecName "kube-api-access-tn4bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:24.684898 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.684862 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651c823f-662b-404a-9374-c0f641f322df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651c823f-662b-404a-9374-c0f641f322df-plugins-config-volume\") pod \"651c823f-662b-404a-9374-c0f641f322df\" (UID: \"651c823f-662b-404a-9374-c0f641f322df\") " Oct 30 01:25:24.685073 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.685007 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651c823f-662b-404a-9374-c0f641f322df-kube-api-access-b7s77\" (UniqueName: \"kubernetes.io/projected/651c823f-662b-404a-9374-c0f641f322df-kube-api-access-b7s77\") pod \"651c823f-662b-404a-9374-c0f641f322df\" (UID: \"651c823f-662b-404a-9374-c0f641f322df\") " Oct 30 01:25:24.685218 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.685206 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff872648-3c7b-4b11-90b1-8d34939b17ce-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:24.685261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.685220 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4bc\" (UniqueName: \"kubernetes.io/projected/ff872648-3c7b-4b11-90b1-8d34939b17ce-kube-api-access-tn4bc\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:24.685261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.685233 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651c823f-662b-404a-9374-c0f641f322df-plugins-config-volume" pod "651c823f-662b-404a-9374-c0f641f322df" (UID: "651c823f-662b-404a-9374-c0f641f322df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:24.687071 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.687044 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651c823f-662b-404a-9374-c0f641f322df-kube-api-access-b7s77" pod "651c823f-662b-404a-9374-c0f641f322df" (UID: "651c823f-662b-404a-9374-c0f641f322df"). InnerVolumeSpecName "kube-api-access-b7s77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:24.786025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.785940 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7s77\" (UniqueName: \"kubernetes.io/projected/651c823f-662b-404a-9374-c0f641f322df-kube-api-access-b7s77\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:24.786025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:24.785980 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651c823f-662b-404a-9374-c0f641f322df-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:25.464251 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.464220 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" Oct 30 01:25:25.464737 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.464217 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w" event={"ID":"651c823f-662b-404a-9374-c0f641f322df","Type":"ContainerDied","Data":"b979e76e91c020b55b8ea924369173f9ebdc1ec7867d4e8203c6b383481cc5ba"} Oct 30 01:25:25.464737 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.464359 4927 scope.go:122] "RemoveContainer" containerID="f1feecee5b81e8bdf6a7c5577a64235bffd47eb685ee2afc68fddf36812cc655" Oct 30 01:25:25.465750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.465731 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" Oct 30 01:25:25.465826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.465765 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v" event={"ID":"ff872648-3c7b-4b11-90b1-8d34939b17ce","Type":"ContainerDied","Data":"a3fb885c99b955ba789cab59c3810d4956db84f11294301e992747cc28d9a7e7"} Oct 30 01:25:25.479494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.479474 4927 scope.go:122] "RemoveContainer" containerID="a2e19378d9e052fb44629c34170b01047012a84f959b6ba676d291ed2cc0af39" Oct 30 01:25:25.528968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.528932 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w"] Oct 30 01:25:25.539693 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.539665 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-95492e-c44db6c87-vdr9w"] Oct 30 01:25:25.572267 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.572240 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651c823f-662b-404a-9374-c0f641f322df" path="/var/lib/kubelet/pods/651c823f-662b-404a-9374-c0f641f322df/volumes" Oct 30 01:25:25.572559 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.572544 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v"] Oct 30 01:25:25.572624 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:25.572588 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-95492e-6cbcb58bc4-qdj6v"] Oct 30 01:25:27.570860 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:27.570823 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff872648-3c7b-4b11-90b1-8d34939b17ce" path="/var/lib/kubelet/pods/ff872648-3c7b-4b11-90b1-8d34939b17ce/volumes" Oct 30 01:25:28.426584 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.426536 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" podUID="d35f1d3f-c7b4-46c0-b342-4d97da8aff8d" containerName="payload-pre-processing" containerID="cri-o://f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71" gracePeriod=25 Oct 30 01:25:28.706762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.706733 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:25:28.828535 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.828497 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-kube-api-access-z5776\" (UniqueName: \"kubernetes.io/projected/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-kube-api-access-z5776\") pod \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\" (UID: \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\") " Oct 30 01:25:28.828752 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.828553 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-plugins-config-volume\") pod \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\" (UID: \"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d\") " Oct 30 01:25:28.829124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.829096 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-plugins-config-volume" pod "d35f1d3f-c7b4-46c0-b342-4d97da8aff8d" (UID: "d35f1d3f-c7b4-46c0-b342-4d97da8aff8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:28.830954 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.830934 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-kube-api-access-z5776" pod "d35f1d3f-c7b4-46c0-b342-4d97da8aff8d" (UID: "d35f1d3f-c7b4-46c0-b342-4d97da8aff8d"). InnerVolumeSpecName "kube-api-access-z5776". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:28.862306 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.862259 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd"] Oct 30 01:25:28.865153 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.865113 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-16c1bb-qhvqd"] Oct 30 01:25:28.929849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.929819 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5776\" (UniqueName: \"kubernetes.io/projected/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-kube-api-access-z5776\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:28.929849 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:28.929844 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:29.494432 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.494393 4927 generic.go:356] "Generic (PLEG): container finished" podID="d35f1d3f-c7b4-46c0-b342-4d97da8aff8d" containerID="f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71" exitCode=0 Oct 30 01:25:29.494675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.494448 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" Oct 30 01:25:29.494675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.494482 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" event={"ID":"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d","Type":"ContainerDied","Data":"f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71"} Oct 30 01:25:29.494675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.494663 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b" event={"ID":"d35f1d3f-c7b4-46c0-b342-4d97da8aff8d","Type":"ContainerDied","Data":"d62eee9e6425ebabe3d0251da6736162971955334ed5fbabea40897715dd5b88"} Oct 30 01:25:29.494874 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.494734 4927 scope.go:122] "RemoveContainer" containerID="f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71" Oct 30 01:25:29.506976 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.506958 4927 scope.go:122] "RemoveContainer" containerID="f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71" Oct 30 01:25:29.507303 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:29.507279 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71\": container with ID starting with f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71 not found: ID does not exist" containerID="f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71" Oct 30 01:25:29.507386 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.507320 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71"} err="failed to get container status \"f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71\": rpc error: code = NotFound desc = could not find container \"f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71\": container with ID starting with f8dd8ceafbb5b93b086eb8dfe0718c23b22c2d0d09be93841a5b76acecc51c71 not found: ID does not exist" Oct 30 01:25:29.543890 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.543862 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b"] Oct 30 01:25:29.557816 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.556983 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d01638-74f77f4787-j758b"] Oct 30 01:25:29.572230 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.572197 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35f1d3f-c7b4-46c0-b342-4d97da8aff8d" path="/var/lib/kubelet/pods/d35f1d3f-c7b4-46c0-b342-4d97da8aff8d/volumes" Oct 30 01:25:29.572812 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:29.572794 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51d8b20-a2bc-47d5-befa-55eb443e7ba2" path="/var/lib/kubelet/pods/e51d8b20-a2bc-47d5-befa-55eb443e7ba2/volumes" Oct 30 01:25:34.441744 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:34.441700 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:25:37.107319 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:37.107270 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx"] Oct 30 01:25:37.107857 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:37.107561 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" containerID="cri-o://5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d" gracePeriod=30 Oct 30 01:25:41.581957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:41.581922 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv"] Oct 30 01:25:41.834495 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:41.834406 4927 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:25:41.834495 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:41.834477 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:25:42.360274 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.360248 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:25:42.466690 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.466656 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-podinfo\") pod \"9a54ef9a-e71a-45e3-af09-92b766b0432e\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " Oct 30 01:25:42.466885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.466708 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-kube-api-access-fwphx\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-kube-api-access-fwphx\") pod \"9a54ef9a-e71a-45e3-af09-92b766b0432e\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " Oct 30 01:25:42.466885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.466751 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-envoy\") pod \"9a54ef9a-e71a-45e3-af09-92b766b0432e\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " Oct 30 01:25:42.466885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.466790 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-socket\") pod \"9a54ef9a-e71a-45e3-af09-92b766b0432e\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " Oct 30 01:25:42.466885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.466835 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a54ef9a-e71a-45e3-af09-92b766b0432e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a54ef9a-e71a-45e3-af09-92b766b0432e-istiod-ca-cert\") pod \"9a54ef9a-e71a-45e3-af09-92b766b0432e\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " Oct 30 01:25:42.467123 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.466889 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-token\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-token\") pod \"9a54ef9a-e71a-45e3-af09-92b766b0432e\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " Oct 30 01:25:42.467123 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.466944 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-certs\") pod \"9a54ef9a-e71a-45e3-af09-92b766b0432e\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " Oct 30 01:25:42.467123 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.466989 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-data\") pod \"9a54ef9a-e71a-45e3-af09-92b766b0432e\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " Oct 30 01:25:42.467123 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.467029 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-credential-socket\") pod \"9a54ef9a-e71a-45e3-af09-92b766b0432e\" (UID: \"9a54ef9a-e71a-45e3-af09-92b766b0432e\") " Oct 30 01:25:42.467359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.467198 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-socket" pod "9a54ef9a-e71a-45e3-af09-92b766b0432e" (UID: "9a54ef9a-e71a-45e3-af09-92b766b0432e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:42.467633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.467421 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a54ef9a-e71a-45e3-af09-92b766b0432e-istiod-ca-cert" pod "9a54ef9a-e71a-45e3-af09-92b766b0432e" (UID: "9a54ef9a-e71a-45e3-af09-92b766b0432e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:42.467633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.467444 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:42.467633 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.467473 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-credential-socket" pod "9a54ef9a-e71a-45e3-af09-92b766b0432e" (UID: "9a54ef9a-e71a-45e3-af09-92b766b0432e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:42.467826 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.467729 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-data" pod "9a54ef9a-e71a-45e3-af09-92b766b0432e" (UID: "9a54ef9a-e71a-45e3-af09-92b766b0432e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:42.467934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.467909 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-certs" pod "9a54ef9a-e71a-45e3-af09-92b766b0432e" (UID: "9a54ef9a-e71a-45e3-af09-92b766b0432e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:42.469101 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.469081 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-kube-api-access-fwphx" pod "9a54ef9a-e71a-45e3-af09-92b766b0432e" (UID: "9a54ef9a-e71a-45e3-af09-92b766b0432e"). InnerVolumeSpecName "kube-api-access-fwphx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:42.469349 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.469322 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-token" pod "9a54ef9a-e71a-45e3-af09-92b766b0432e" (UID: "9a54ef9a-e71a-45e3-af09-92b766b0432e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:42.469439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.469411 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-envoy" pod "9a54ef9a-e71a-45e3-af09-92b766b0432e" (UID: "9a54ef9a-e71a-45e3-af09-92b766b0432e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:42.469439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.469421 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-podinfo" pod "9a54ef9a-e71a-45e3-af09-92b766b0432e" (UID: "9a54ef9a-e71a-45e3-af09-92b766b0432e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:42.568261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.568222 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a54ef9a-e71a-45e3-af09-92b766b0432e-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:42.568261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.568257 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:42.568261 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.568267 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:42.568506 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.568275 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:42.568506 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.568283 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:42.568506 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.568292 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:42.568506 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.568301 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwphx\" (UniqueName: \"kubernetes.io/projected/9a54ef9a-e71a-45e3-af09-92b766b0432e-kube-api-access-fwphx\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:42.568506 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.568314 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a54ef9a-e71a-45e3-af09-92b766b0432e-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:42.587144 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.587109 4927 generic.go:356] "Generic (PLEG): container finished" podID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerID="5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d" exitCode=0 Oct 30 01:25:42.587641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.587178 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" Oct 30 01:25:42.587641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.587197 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" event={"ID":"9a54ef9a-e71a-45e3-af09-92b766b0432e","Type":"ContainerDied","Data":"5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d"} Oct 30 01:25:42.587641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.587270 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx" event={"ID":"9a54ef9a-e71a-45e3-af09-92b766b0432e","Type":"ContainerDied","Data":"6027aad010494efb06703c8768996f2877c37114386505840c107fc8b1671ad1"} Oct 30 01:25:42.587641 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.587313 4927 scope.go:122] "RemoveContainer" containerID="5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d" Oct 30 01:25:42.598160 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.598140 4927 scope.go:122] "RemoveContainer" containerID="bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f" Oct 30 01:25:42.608536 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.608517 4927 scope.go:122] "RemoveContainer" containerID="5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d" Oct 30 01:25:42.608876 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:42.608853 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d\": container with ID starting with 5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d not found: ID does not exist" containerID="5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d" Oct 30 01:25:42.608957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.608885 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d"} err="failed to get container status \"5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d\": rpc error: code = NotFound desc = could not find container \"5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d\": container with ID starting with 5f172ad3e2605a008af7bebaeaff97c226bd2c8f603c12f1cb9924f6c8caeb8d not found: ID does not exist" Oct 30 01:25:42.608957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.608904 4927 scope.go:122] "RemoveContainer" containerID="bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f" Oct 30 01:25:42.609177 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:42.609159 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f\": container with ID starting with bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f not found: ID does not exist" containerID="bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f" Oct 30 01:25:42.609227 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.609185 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f"} err="failed to get container status \"bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f\": rpc error: code = NotFound desc = could not find container \"bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f\": container with ID starting with bf1b0f6c8e159d357798abad60543384cdec3a410afdee9f36d5c53a282dbb0f not found: ID does not exist" Oct 30 01:25:42.640172 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.640125 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx"] Oct 30 01:25:42.648118 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:42.648086 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d01638-openshift-default-884b45857-7kkmx"] Oct 30 01:25:43.570790 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:43.570757 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" path="/var/lib/kubelet/pods/9a54ef9a-e71a-45e3-af09-92b766b0432e/volumes" Oct 30 01:25:47.177076 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:47.177039 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m"] Oct 30 01:25:47.207658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:47.207626 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr"] Oct 30 01:25:47.890958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:47.890922 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff"] Oct 30 01:25:52.178299 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:52.178238 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" podUID="c4670c74-4555-438a-bdb2-b8d9eafb1a19" containerName="payload-processing" containerID="cri-o://c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5" gracePeriod=25 Oct 30 01:25:52.208850 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:52.208785 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" podUID="3540ce90-64ec-436a-9c82-21dd2cc2901a" containerName="payload-pre-processing" containerID="cri-o://af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b" gracePeriod=25 Oct 30 01:25:52.892157 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:52.892082 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" podUID="61b4278f-a348-4a48-8cfa-53508163347f" containerName="payload-processing" containerID="cri-o://187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c" gracePeriod=25 Oct 30 01:25:53.015431 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:25:53.015406 4927 logging.go:55] [core] [Channel #13392 SubChannel #13393]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.178:50052", ServerName: "10.132.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.178:50052: connect: connection refused" Oct 30 01:25:53.161439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.161414 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:25:53.278191 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.278159 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b4278f-a348-4a48-8cfa-53508163347f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b4278f-a348-4a48-8cfa-53508163347f-plugins-config-volume\") pod \"61b4278f-a348-4a48-8cfa-53508163347f\" (UID: \"61b4278f-a348-4a48-8cfa-53508163347f\") " Oct 30 01:25:53.278720 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.278212 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b4278f-a348-4a48-8cfa-53508163347f-kube-api-access-62bm9\" (UniqueName: \"kubernetes.io/projected/61b4278f-a348-4a48-8cfa-53508163347f-kube-api-access-62bm9\") pod \"61b4278f-a348-4a48-8cfa-53508163347f\" (UID: \"61b4278f-a348-4a48-8cfa-53508163347f\") " Oct 30 01:25:53.278720 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.278640 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b4278f-a348-4a48-8cfa-53508163347f-plugins-config-volume" pod "61b4278f-a348-4a48-8cfa-53508163347f" (UID: "61b4278f-a348-4a48-8cfa-53508163347f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:53.280530 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.280499 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b4278f-a348-4a48-8cfa-53508163347f-kube-api-access-62bm9" pod "61b4278f-a348-4a48-8cfa-53508163347f" (UID: "61b4278f-a348-4a48-8cfa-53508163347f"). InnerVolumeSpecName "kube-api-access-62bm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:53.312729 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.312688 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" podUID="3540ce90-64ec-436a-9c82-21dd2cc2901a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:53.312865 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.312719 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" podUID="c4670c74-4555-438a-bdb2-b8d9eafb1a19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:25:53.380234 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.380188 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b4278f-a348-4a48-8cfa-53508163347f-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:53.380234 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.380226 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62bm9\" (UniqueName: \"kubernetes.io/projected/61b4278f-a348-4a48-8cfa-53508163347f-kube-api-access-62bm9\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:53.653368 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.653328 4927 generic.go:356] "Generic (PLEG): container finished" podID="61b4278f-a348-4a48-8cfa-53508163347f" containerID="187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c" exitCode=0 Oct 30 01:25:53.653566 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.653379 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" Oct 30 01:25:53.653566 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.653411 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" event={"ID":"61b4278f-a348-4a48-8cfa-53508163347f","Type":"ContainerDied","Data":"187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c"} Oct 30 01:25:53.653566 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.653465 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" event={"ID":"61b4278f-a348-4a48-8cfa-53508163347f","Type":"ContainerDied","Data":"0ed38482c186b66d579496e24b532a2ffb5097488a2383edf4c2a255d84df0f2"} Oct 30 01:25:53.653566 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.653490 4927 scope.go:122] "RemoveContainer" containerID="187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c" Oct 30 01:25:53.663389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.663347 4927 scope.go:122] "RemoveContainer" containerID="187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c" Oct 30 01:25:53.663751 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:53.663727 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c\": container with ID starting with 187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c not found: ID does not exist" containerID="187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c" Oct 30 01:25:53.663833 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.663759 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c"} err="failed to get container status \"187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c\": rpc error: code = NotFound desc = could not find container \"187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c\": container with ID starting with 187117f8197df0e5029b8f1135d8b82dc74cb9b683ed00c71e8026098aa30b3c not found: ID does not exist" Oct 30 01:25:53.706502 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.706467 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff"] Oct 30 01:25:53.717337 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:53.717309 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff"] Oct 30 01:25:54.015154 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:54.015106 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e24a5-5f6c6bcd5d-h5dff" podUID="61b4278f-a348-4a48-8cfa-53508163347f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.178:50052\" within 1s: context deadline exceeded" Oct 30 01:25:55.569914 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:55.569871 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b4278f-a348-4a48-8cfa-53508163347f" path="/var/lib/kubelet/pods/61b4278f-a348-4a48-8cfa-53508163347f/volumes" Oct 30 01:25:56.605813 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:56.605764 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" containerID="cri-o://507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3" gracePeriod=45 Oct 30 01:25:56.874145 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:56.874119 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 01:25:57.017189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.017156 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-home\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-home\") pod \"9c804efc-0218-41c4-9f85-bb362fb167d5\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " Oct 30 01:25:57.017376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.017267 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-tmp-dir\") pod \"9c804efc-0218-41c4-9f85-bb362fb167d5\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " Oct 30 01:25:57.017376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.017311 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-model-cache\") pod \"9c804efc-0218-41c4-9f85-bb362fb167d5\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " Oct 30 01:25:57.017376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.017356 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-dshm\") pod \"9c804efc-0218-41c4-9f85-bb362fb167d5\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " Oct 30 01:25:57.018066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.017423 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c804efc-0218-41c4-9f85-bb362fb167d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c804efc-0218-41c4-9f85-bb362fb167d5-tls-certs\") pod \"9c804efc-0218-41c4-9f85-bb362fb167d5\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " Oct 30 01:25:57.018066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.017473 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c804efc-0218-41c4-9f85-bb362fb167d5-kube-api-access-6wrwr\" (UniqueName: \"kubernetes.io/projected/9c804efc-0218-41c4-9f85-bb362fb167d5-kube-api-access-6wrwr\") pod \"9c804efc-0218-41c4-9f85-bb362fb167d5\" (UID: \"9c804efc-0218-41c4-9f85-bb362fb167d5\") " Oct 30 01:25:57.018066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.017607 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-tmp-dir" pod "9c804efc-0218-41c4-9f85-bb362fb167d5" (UID: "9c804efc-0218-41c4-9f85-bb362fb167d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.018066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.017621 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-home" pod "9c804efc-0218-41c4-9f85-bb362fb167d5" (UID: "9c804efc-0218-41c4-9f85-bb362fb167d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.018066 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.017926 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-model-cache" pod "9c804efc-0218-41c4-9f85-bb362fb167d5" (UID: "9c804efc-0218-41c4-9f85-bb362fb167d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.018428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.018072 4927 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.018428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.018111 4927 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-home\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.018428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.018130 4927 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-tmp-dir\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.020748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.020724 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c804efc-0218-41c4-9f85-bb362fb167d5-tls-certs" pod "9c804efc-0218-41c4-9f85-bb362fb167d5" (UID: "9c804efc-0218-41c4-9f85-bb362fb167d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:57.020748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.020734 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-dshm" pod "9c804efc-0218-41c4-9f85-bb362fb167d5" (UID: "9c804efc-0218-41c4-9f85-bb362fb167d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:57.020876 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.020746 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c804efc-0218-41c4-9f85-bb362fb167d5-kube-api-access-6wrwr" pod "9c804efc-0218-41c4-9f85-bb362fb167d5" (UID: "9c804efc-0218-41c4-9f85-bb362fb167d5"). InnerVolumeSpecName "kube-api-access-6wrwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:57.119009 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.118933 4927 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c804efc-0218-41c4-9f85-bb362fb167d5-dshm\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.119009 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.118964 4927 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c804efc-0218-41c4-9f85-bb362fb167d5-tls-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.119009 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.118976 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wrwr\" (UniqueName: \"kubernetes.io/projected/9c804efc-0218-41c4-9f85-bb362fb167d5-kube-api-access-6wrwr\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:25:57.679608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.679555 4927 generic.go:356] "Generic (PLEG): container finished" podID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerID="507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3" exitCode=0 Oct 30 01:25:57.680150 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.679639 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" event={"ID":"9c804efc-0218-41c4-9f85-bb362fb167d5","Type":"ContainerDied","Data":"507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3"} Oct 30 01:25:57.680304 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.679634 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" Oct 30 01:25:57.680304 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.680172 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv" event={"ID":"9c804efc-0218-41c4-9f85-bb362fb167d5","Type":"ContainerDied","Data":"5664bb00e5385f98728cbe8c7516699be81c5c29a0e83db4010f586b2ad4d682"} Oct 30 01:25:57.680304 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.680194 4927 scope.go:122] "RemoveContainer" containerID="507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3" Oct 30 01:25:57.690104 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.690083 4927 scope.go:122] "RemoveContainer" containerID="507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3" Oct 30 01:25:57.690370 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:25:57.690349 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3\": container with ID starting with 507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3 not found: ID does not exist" containerID="507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3" Oct 30 01:25:57.690410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.690379 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3"} err="failed to get container status \"507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3\": rpc error: code = NotFound desc = could not find container \"507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3\": container with ID starting with 507ddd3278800e300189a6157fb0cd8701830e014ecf57c7e6d9a50ad5262bf3 not found: ID does not exist" Oct 30 01:25:57.728124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.728099 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv"] Oct 30 01:25:57.739275 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:57.739251 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-beb8d5/distinct-2-w1-beb8d5-kserve-5ccb98dbd7-4czsv"] Oct 30 01:25:59.570273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:25:59.570241 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" path="/var/lib/kubelet/pods/9c804efc-0218-41c4-9f85-bb362fb167d5/volumes" Oct 30 01:26:03.312675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:03.312591 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" podUID="3540ce90-64ec-436a-9c82-21dd2cc2901a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:26:03.313056 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:03.312588 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" podUID="c4670c74-4555-438a-bdb2-b8d9eafb1a19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:26:04.441683 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:26:04.441646 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:26:08.334347 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.334302 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86"] Oct 30 01:26:08.540835 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.540768 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr"] Oct 30 01:26:08.541393 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.541351 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" podUID="bbef5382-412e-4b85-989e-d728b15c9c72" containerName="maas-api" containerID="cri-o://2a40aff28eb1a2ac47df36691e66c32d5919f8d05b643ebe1ad3067def0e9618" gracePeriod=30 Oct 30 01:26:08.748805 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.748768 4927 generic.go:356] "Generic (PLEG): container finished" podID="bbef5382-412e-4b85-989e-d728b15c9c72" containerID="2a40aff28eb1a2ac47df36691e66c32d5919f8d05b643ebe1ad3067def0e9618" exitCode=0 Oct 30 01:26:08.748974 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.748841 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" event={"ID":"bbef5382-412e-4b85-989e-d728b15c9c72","Type":"ContainerDied","Data":"2a40aff28eb1a2ac47df36691e66c32d5919f8d05b643ebe1ad3067def0e9618"} Oct 30 01:26:08.795349 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.795320 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 01:26:08.844830 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.844800 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-maas-api-tls\") pod \"bbef5382-412e-4b85-989e-d728b15c9c72\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " Oct 30 01:26:08.845032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.844880 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbef5382-412e-4b85-989e-d728b15c9c72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbef5382-412e-4b85-989e-d728b15c9c72-openshift-service-ca\") pod \"bbef5382-412e-4b85-989e-d728b15c9c72\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " Oct 30 01:26:08.845032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.844925 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbef5382-412e-4b85-989e-d728b15c9c72-kube-api-access-sr444\" (UniqueName: \"kubernetes.io/projected/bbef5382-412e-4b85-989e-d728b15c9c72-kube-api-access-sr444\") pod \"bbef5382-412e-4b85-989e-d728b15c9c72\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " Oct 30 01:26:08.845032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.844959 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-metrics-certs\") pod \"bbef5382-412e-4b85-989e-d728b15c9c72\" (UID: \"bbef5382-412e-4b85-989e-d728b15c9c72\") " Oct 30 01:26:08.845329 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.845298 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbef5382-412e-4b85-989e-d728b15c9c72-openshift-service-ca" pod "bbef5382-412e-4b85-989e-d728b15c9c72" (UID: "bbef5382-412e-4b85-989e-d728b15c9c72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:08.847265 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.847238 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-maas-api-tls" pod "bbef5382-412e-4b85-989e-d728b15c9c72" (UID: "bbef5382-412e-4b85-989e-d728b15c9c72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:08.847529 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.847498 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-metrics-certs" pod "bbef5382-412e-4b85-989e-d728b15c9c72" (UID: "bbef5382-412e-4b85-989e-d728b15c9c72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:08.847672 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.847649 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbef5382-412e-4b85-989e-d728b15c9c72-kube-api-access-sr444" pod "bbef5382-412e-4b85-989e-d728b15c9c72" (UID: "bbef5382-412e-4b85-989e-d728b15c9c72"). InnerVolumeSpecName "kube-api-access-sr444". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:08.949866 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.949778 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:08.950131 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.950097 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbef5382-412e-4b85-989e-d728b15c9c72-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:08.950131 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.950131 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr444\" (UniqueName: \"kubernetes.io/projected/bbef5382-412e-4b85-989e-d728b15c9c72-kube-api-access-sr444\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:08.950399 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:08.950148 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbef5382-412e-4b85-989e-d728b15c9c72-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:09.756354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:09.756322 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" Oct 30 01:26:09.756884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:09.756355 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr" event={"ID":"bbef5382-412e-4b85-989e-d728b15c9c72","Type":"ContainerDied","Data":"9062c397a9b5cb97cd5cfe486f84be7e604fc026a9d42e482550ac6adb34b010"} Oct 30 01:26:09.756884 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:09.756419 4927 scope.go:122] "RemoveContainer" containerID="2a40aff28eb1a2ac47df36691e66c32d5919f8d05b643ebe1ad3067def0e9618" Oct 30 01:26:09.810700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:09.810669 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr"] Oct 30 01:26:09.817132 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:09.817107 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-beb8d5-bd885c7d9-x6lrr"] Oct 30 01:26:10.805308 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.805268 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn"] Oct 30 01:26:10.805988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.805964 4927 cpu_manager.go:414] "containerMap: removing container" podUID="ff872648-3c7b-4b11-90b1-8d34939b17ce" containerName="payload-processing" Oct 30 01:26:10.806065 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.805992 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff872648-3c7b-4b11-90b1-8d34939b17ce" containerName="payload-processing" Oct 30 01:26:10.806065 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806017 4927 cpu_manager.go:414] "containerMap: removing container" podUID="bbef5382-412e-4b85-989e-d728b15c9c72" containerName="maas-api" Oct 30 01:26:10.806065 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806026 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbef5382-412e-4b85-989e-d728b15c9c72" containerName="maas-api" Oct 30 01:26:10.806065 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806038 4927 cpu_manager.go:414] "containerMap: removing container" podUID="651c823f-662b-404a-9374-c0f641f322df" containerName="payload-pre-processing" Oct 30 01:26:10.806065 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806047 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651c823f-662b-404a-9374-c0f641f322df" containerName="payload-pre-processing" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806069 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806077 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806089 4927 cpu_manager.go:414] "containerMap: removing container" podUID="e51d8b20-a2bc-47d5-befa-55eb443e7ba2" containerName="revoke-keys" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806097 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51d8b20-a2bc-47d5-befa-55eb443e7ba2" containerName="revoke-keys" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806111 4927 cpu_manager.go:414] "containerMap: removing container" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806118 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806133 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806140 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806159 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806166 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806179 4927 cpu_manager.go:414] "containerMap: removing container" podUID="d35f1d3f-c7b4-46c0-b342-4d97da8aff8d" containerName="payload-pre-processing" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806187 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35f1d3f-c7b4-46c0-b342-4d97da8aff8d" containerName="payload-pre-processing" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806205 4927 cpu_manager.go:414] "containerMap: removing container" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806213 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806225 4927 cpu_manager.go:414] "containerMap: removing container" podUID="61b4278f-a348-4a48-8cfa-53508163347f" containerName="payload-processing" Oct 30 01:26:10.806334 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806231 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b4278f-a348-4a48-8cfa-53508163347f" containerName="payload-processing" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806350 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff872648-3c7b-4b11-90b1-8d34939b17ce" containerName="payload-processing" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806364 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbef5382-412e-4b85-989e-d728b15c9c72" containerName="maas-api" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806373 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="651c823f-662b-404a-9374-c0f641f322df" containerName="payload-pre-processing" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806383 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806394 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51d8b20-a2bc-47d5-befa-55eb443e7ba2" containerName="revoke-keys" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806404 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="1038975e-5fd3-4674-9374-e4a46799b7ff" containerName="istio-proxy" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806416 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806425 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a54ef9a-e71a-45e3-af09-92b766b0432e" containerName="istio-proxy" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806436 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35f1d3f-c7b4-46c0-b342-4d97da8aff8d" containerName="payload-pre-processing" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806445 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c804efc-0218-41c4-9f85-bb362fb167d5" containerName="main" Oct 30 01:26:10.807176 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.806453 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b4278f-a348-4a48-8cfa-53508163347f" containerName="payload-processing" Oct 30 01:26:10.811701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.811679 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:10.827372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.827344 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn"] Oct 30 01:26:10.867983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.867942 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcprx\" (UniqueName: \"kubernetes.io/projected/424cef29-a2d3-411d-aab4-a379e0e740a5-kube-api-access-kcprx\") pod \"maas-api-revoke-sub-8ffee52f-g27dn\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:10.868153 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.867991 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ffee52f-g27dn\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:10.868200 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.868181 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ffee52f-g27dn\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:10.901451 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.901423 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:26:10.901614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.901502 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\"" Oct 30 01:26:10.969723 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.969679 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ffee52f-g27dn\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:10.969911 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.969742 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcprx\" (UniqueName: \"kubernetes.io/projected/424cef29-a2d3-411d-aab4-a379e0e740a5-kube-api-access-kcprx\") pod \"maas-api-revoke-sub-8ffee52f-g27dn\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:10.969911 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.969774 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ffee52f-g27dn\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:10.970357 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.970331 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ffee52f-g27dn\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:10.972206 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.972183 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ffee52f-g27dn\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:10.992308 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:10.992281 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcprx\" (UniqueName: \"kubernetes.io/projected/424cef29-a2d3-411d-aab4-a379e0e740a5-kube-api-access-kcprx\") pod \"maas-api-revoke-sub-8ffee52f-g27dn\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:11.125711 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:11.125677 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:11.427250 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:11.427220 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn"] Oct 30 01:26:11.429699 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:26:11.429669 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424cef29_a2d3_411d_aab4_a379e0e740a5.slice/crio-fe91ee665e27e5b64d1f5f4ec7f421335dc38046281e706c7e090cf1ae41e32f WatchSource:0}: Error finding container fe91ee665e27e5b64d1f5f4ec7f421335dc38046281e706c7e090cf1ae41e32f: Status 404 returned error can't find the container with id fe91ee665e27e5b64d1f5f4ec7f421335dc38046281e706c7e090cf1ae41e32f Oct 30 01:26:11.569932 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:11.569898 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbef5382-412e-4b85-989e-d728b15c9c72" path="/var/lib/kubelet/pods/bbef5382-412e-4b85-989e-d728b15c9c72/volumes" Oct 30 01:26:11.770997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:11.770961 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" event={"ID":"424cef29-a2d3-411d-aab4-a379e0e740a5","Type":"ContainerStarted","Data":"91d23814acf449cd94de5bb8e4c1332b638c12e0bc22f683ef2d3113904b8f2f"} Oct 30 01:26:11.771170 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:11.771018 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" event={"ID":"424cef29-a2d3-411d-aab4-a379e0e740a5","Type":"ContainerStarted","Data":"fe91ee665e27e5b64d1f5f4ec7f421335dc38046281e706c7e090cf1ae41e32f"} Oct 30 01:26:11.839450 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:11.839393 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" podStartSLOduration=1.839372654 podStartE2EDuration="1.839372654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:11.837839833 +0000 UTC m=+3110.409766054" watchObservedRunningTime="2026-10-30 01:26:11.839372654 +0000 UTC m=+3110.411298877" Oct 30 01:26:12.459027 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.459003 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 01:26:12.484306 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.484282 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 01:26:12.487459 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.487437 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4670c74-4555-438a-bdb2-b8d9eafb1a19-kube-api-access-dhn7v\" (UniqueName: \"kubernetes.io/projected/c4670c74-4555-438a-bdb2-b8d9eafb1a19-kube-api-access-dhn7v\") pod \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\" (UID: \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\") " Oct 30 01:26:12.487631 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.487613 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4670c74-4555-438a-bdb2-b8d9eafb1a19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4670c74-4555-438a-bdb2-b8d9eafb1a19-plugins-config-volume\") pod \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\" (UID: \"c4670c74-4555-438a-bdb2-b8d9eafb1a19\") " Oct 30 01:26:12.487922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.487902 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4670c74-4555-438a-bdb2-b8d9eafb1a19-plugins-config-volume" pod "c4670c74-4555-438a-bdb2-b8d9eafb1a19" (UID: "c4670c74-4555-438a-bdb2-b8d9eafb1a19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:12.489433 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.489414 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4670c74-4555-438a-bdb2-b8d9eafb1a19-kube-api-access-dhn7v" pod "c4670c74-4555-438a-bdb2-b8d9eafb1a19" (UID: "c4670c74-4555-438a-bdb2-b8d9eafb1a19"). InnerVolumeSpecName "kube-api-access-dhn7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:12.588809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.588777 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3540ce90-64ec-436a-9c82-21dd2cc2901a-kube-api-access-brn9p\" (UniqueName: \"kubernetes.io/projected/3540ce90-64ec-436a-9c82-21dd2cc2901a-kube-api-access-brn9p\") pod \"3540ce90-64ec-436a-9c82-21dd2cc2901a\" (UID: \"3540ce90-64ec-436a-9c82-21dd2cc2901a\") " Oct 30 01:26:12.588809 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.588814 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3540ce90-64ec-436a-9c82-21dd2cc2901a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3540ce90-64ec-436a-9c82-21dd2cc2901a-plugins-config-volume\") pod \"3540ce90-64ec-436a-9c82-21dd2cc2901a\" (UID: \"3540ce90-64ec-436a-9c82-21dd2cc2901a\") " Oct 30 01:26:12.589131 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.589114 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4670c74-4555-438a-bdb2-b8d9eafb1a19-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:12.589131 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.589132 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhn7v\" (UniqueName: \"kubernetes.io/projected/c4670c74-4555-438a-bdb2-b8d9eafb1a19-kube-api-access-dhn7v\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:12.589320 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.589295 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3540ce90-64ec-436a-9c82-21dd2cc2901a-plugins-config-volume" pod "3540ce90-64ec-436a-9c82-21dd2cc2901a" (UID: "3540ce90-64ec-436a-9c82-21dd2cc2901a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:12.590973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.590947 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3540ce90-64ec-436a-9c82-21dd2cc2901a-kube-api-access-brn9p" pod "3540ce90-64ec-436a-9c82-21dd2cc2901a" (UID: "3540ce90-64ec-436a-9c82-21dd2cc2901a"). InnerVolumeSpecName "kube-api-access-brn9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:12.689997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.689910 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brn9p\" (UniqueName: \"kubernetes.io/projected/3540ce90-64ec-436a-9c82-21dd2cc2901a-kube-api-access-brn9p\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:12.689997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.689938 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3540ce90-64ec-436a-9c82-21dd2cc2901a-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:12.779216 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.779182 4927 generic.go:356] "Generic (PLEG): container finished" podID="c4670c74-4555-438a-bdb2-b8d9eafb1a19" containerID="c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5" exitCode=0 Oct 30 01:26:12.779422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.779229 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" Oct 30 01:26:12.779422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.779280 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" event={"ID":"c4670c74-4555-438a-bdb2-b8d9eafb1a19","Type":"ContainerDied","Data":"c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5"} Oct 30 01:26:12.779422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.779345 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m" event={"ID":"c4670c74-4555-438a-bdb2-b8d9eafb1a19","Type":"ContainerDied","Data":"c2a878129f4d373aa395014524ffcdf49308af9b2d98dab4977a345156ced173"} Oct 30 01:26:12.779422 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.779377 4927 scope.go:122] "RemoveContainer" containerID="c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5" Oct 30 01:26:12.781218 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.781189 4927 generic.go:356] "Generic (PLEG): container finished" podID="424cef29-a2d3-411d-aab4-a379e0e740a5" containerID="91d23814acf449cd94de5bb8e4c1332b638c12e0bc22f683ef2d3113904b8f2f" exitCode=0 Oct 30 01:26:12.781325 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.781233 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" event={"ID":"424cef29-a2d3-411d-aab4-a379e0e740a5","Type":"ContainerDied","Data":"91d23814acf449cd94de5bb8e4c1332b638c12e0bc22f683ef2d3113904b8f2f"} Oct 30 01:26:12.782934 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.782909 4927 generic.go:356] "Generic (PLEG): container finished" podID="3540ce90-64ec-436a-9c82-21dd2cc2901a" containerID="af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b" exitCode=0 Oct 30 01:26:12.783037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.782948 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" Oct 30 01:26:12.783037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.782951 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" event={"ID":"3540ce90-64ec-436a-9c82-21dd2cc2901a","Type":"ContainerDied","Data":"af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b"} Oct 30 01:26:12.783037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.782991 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr" event={"ID":"3540ce90-64ec-436a-9c82-21dd2cc2901a","Type":"ContainerDied","Data":"32042c5e6d7e9a01f05a89acd4c3f29539e672c48117b2e28b4fe100d7ee4bd3"} Oct 30 01:26:12.790505 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.790482 4927 scope.go:122] "RemoveContainer" containerID="c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5" Oct 30 01:26:12.790866 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:26:12.790846 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5\": container with ID starting with c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5 not found: ID does not exist" containerID="c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5" Oct 30 01:26:12.790973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.790873 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5"} err="failed to get container status \"c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5\": rpc error: code = NotFound desc = could not find container \"c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5\": container with ID starting with c99715d92ce844066e08b7b35303961c708383f226d7d8a7dd089f61a8a5d6b5 not found: ID does not exist" Oct 30 01:26:12.790973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.790890 4927 scope.go:122] "RemoveContainer" containerID="af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b" Oct 30 01:26:12.799453 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.799437 4927 scope.go:122] "RemoveContainer" containerID="af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b" Oct 30 01:26:12.800684 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:26:12.800562 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b\": container with ID starting with af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b not found: ID does not exist" containerID="af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b" Oct 30 01:26:12.800756 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.800697 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b"} err="failed to get container status \"af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b\": rpc error: code = NotFound desc = could not find container \"af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b\": container with ID starting with af70756fd94e3e40de23b3d6923c0e2f49bbc237973538557ccd9c095442b16b not found: ID does not exist" Oct 30 01:26:12.829863 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.829842 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr"] Oct 30 01:26:12.835880 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.835856 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-beb8d5-9bf795695-kmhxr"] Oct 30 01:26:12.907149 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.907118 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m"] Oct 30 01:26:12.911504 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:12.911473 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-beb8d5-cb6dcb8cb-cx87m"] Oct 30 01:26:13.335456 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:13.335362 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" podUID="74d6a3b2-8f58-435c-9591-a6d5db1069ce" containerName="payload-processing" containerID="cri-o://6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6" gracePeriod=25 Oct 30 01:26:13.570124 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:13.570091 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3540ce90-64ec-436a-9c82-21dd2cc2901a" path="/var/lib/kubelet/pods/3540ce90-64ec-436a-9c82-21dd2cc2901a/volumes" Oct 30 01:26:13.570501 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:13.570485 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4670c74-4555-438a-bdb2-b8d9eafb1a19" path="/var/lib/kubelet/pods/c4670c74-4555-438a-bdb2-b8d9eafb1a19/volumes" Oct 30 01:26:13.954273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:13.954247 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:14.002848 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.002820 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-xks-ca\") pod \"424cef29-a2d3-411d-aab4-a379e0e740a5\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " Oct 30 01:26:14.003016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.002862 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-openshift-ca\") pod \"424cef29-a2d3-411d-aab4-a379e0e740a5\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " Oct 30 01:26:14.003016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.002916 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424cef29-a2d3-411d-aab4-a379e0e740a5-kube-api-access-kcprx\" (UniqueName: \"kubernetes.io/projected/424cef29-a2d3-411d-aab4-a379e0e740a5-kube-api-access-kcprx\") pod \"424cef29-a2d3-411d-aab4-a379e0e740a5\" (UID: \"424cef29-a2d3-411d-aab4-a379e0e740a5\") " Oct 30 01:26:14.003376 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.003342 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-openshift-ca" pod "424cef29-a2d3-411d-aab4-a379e0e740a5" (UID: "424cef29-a2d3-411d-aab4-a379e0e740a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:14.005295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.005261 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-xks-ca" pod "424cef29-a2d3-411d-aab4-a379e0e740a5" (UID: "424cef29-a2d3-411d-aab4-a379e0e740a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:14.005716 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.005692 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424cef29-a2d3-411d-aab4-a379e0e740a5-kube-api-access-kcprx" pod "424cef29-a2d3-411d-aab4-a379e0e740a5" (UID: "424cef29-a2d3-411d-aab4-a379e0e740a5"). InnerVolumeSpecName "kube-api-access-kcprx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:14.103868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.103793 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcprx\" (UniqueName: \"kubernetes.io/projected/424cef29-a2d3-411d-aab4-a379e0e740a5-kube-api-access-kcprx\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:14.103868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.103822 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-xks-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:14.103868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.103831 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424cef29-a2d3-411d-aab4-a379e0e740a5-maas-api-openshift-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:14.797592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.797544 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" Oct 30 01:26:14.797592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.797565 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn" event={"ID":"424cef29-a2d3-411d-aab4-a379e0e740a5","Type":"ContainerDied","Data":"fe91ee665e27e5b64d1f5f4ec7f421335dc38046281e706c7e090cf1ae41e32f"} Oct 30 01:26:14.797815 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:14.797623 4927 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe91ee665e27e5b64d1f5f4ec7f421335dc38046281e706c7e090cf1ae41e32f" Oct 30 01:26:18.405792 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:18.405748 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" podUID="74d6a3b2-8f58-435c-9591-a6d5db1069ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:26:28.404606 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:28.404544 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" podUID="74d6a3b2-8f58-435c-9591-a6d5db1069ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:26:33.594882 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.594819 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 01:26:33.694913 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.694882 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d6a3b2-8f58-435c-9591-a6d5db1069ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d6a3b2-8f58-435c-9591-a6d5db1069ce-plugins-config-volume\") pod \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\" (UID: \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\") " Oct 30 01:26:33.695107 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.694975 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d6a3b2-8f58-435c-9591-a6d5db1069ce-kube-api-access-vb7mc\" (UniqueName: \"kubernetes.io/projected/74d6a3b2-8f58-435c-9591-a6d5db1069ce-kube-api-access-vb7mc\") pod \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\" (UID: \"74d6a3b2-8f58-435c-9591-a6d5db1069ce\") " Oct 30 01:26:33.695289 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.695260 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d6a3b2-8f58-435c-9591-a6d5db1069ce-plugins-config-volume" pod "74d6a3b2-8f58-435c-9591-a6d5db1069ce" (UID: "74d6a3b2-8f58-435c-9591-a6d5db1069ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:33.697340 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.697307 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d6a3b2-8f58-435c-9591-a6d5db1069ce-kube-api-access-vb7mc" pod "74d6a3b2-8f58-435c-9591-a6d5db1069ce" (UID: "74d6a3b2-8f58-435c-9591-a6d5db1069ce"). InnerVolumeSpecName "kube-api-access-vb7mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:33.796797 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.796737 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d6a3b2-8f58-435c-9591-a6d5db1069ce-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:33.796797 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.796775 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb7mc\" (UniqueName: \"kubernetes.io/projected/74d6a3b2-8f58-435c-9591-a6d5db1069ce-kube-api-access-vb7mc\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:33.907039 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.906999 4927 generic.go:356] "Generic (PLEG): container finished" podID="74d6a3b2-8f58-435c-9591-a6d5db1069ce" containerID="6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6" exitCode=0 Oct 30 01:26:33.907220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.907064 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" Oct 30 01:26:33.907220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.907051 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" event={"ID":"74d6a3b2-8f58-435c-9591-a6d5db1069ce","Type":"ContainerDied","Data":"6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6"} Oct 30 01:26:33.907220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.907188 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86" event={"ID":"74d6a3b2-8f58-435c-9591-a6d5db1069ce","Type":"ContainerDied","Data":"92fb039498eacbca9d110eded633a1f32806df18532755ad7c7344c5a7c5d150"} Oct 30 01:26:33.907220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.907216 4927 scope.go:122] "RemoveContainer" containerID="6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6" Oct 30 01:26:33.917365 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.917345 4927 scope.go:122] "RemoveContainer" containerID="6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6" Oct 30 01:26:33.917652 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:26:33.917619 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6\": container with ID starting with 6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6 not found: ID does not exist" containerID="6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6" Oct 30 01:26:33.917712 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.917660 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6"} err="failed to get container status \"6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6\": rpc error: code = NotFound desc = could not find container \"6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6\": container with ID starting with 6a74462521455accb914ff0db95e325d108461cca4dc25006eef71f17f5f33f6 not found: ID does not exist" Oct 30 01:26:33.958762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.958732 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86"] Oct 30 01:26:33.969061 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:33.969009 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-89a676-5679c8db-2tb86"] Oct 30 01:26:34.442352 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:26:34.442312 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:26:35.570060 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:35.570026 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d6a3b2-8f58-435c-9591-a6d5db1069ce" path="/var/lib/kubelet/pods/74d6a3b2-8f58-435c-9591-a6d5db1069ce/volumes" Oct 30 01:26:38.463750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.463705 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f6db8855d-hlvbv"] Oct 30 01:26:38.464280 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.463971 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-f6db8855d-hlvbv" podUID="429f70ee-9170-474d-92be-aef90339d241" containerName="manager" containerID="cri-o://c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c" gracePeriod=30 Oct 30 01:26:38.724109 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.724040 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 01:26:38.845647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.845605 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-metrics-certs\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-metrics-certs\") pod \"429f70ee-9170-474d-92be-aef90339d241\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " Oct 30 01:26:38.845827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.845707 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-webhook-cert\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-webhook-cert\") pod \"429f70ee-9170-474d-92be-aef90339d241\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " Oct 30 01:26:38.845827 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.845797 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429f70ee-9170-474d-92be-aef90339d241-kube-api-access-nprk9\" (UniqueName: \"kubernetes.io/projected/429f70ee-9170-474d-92be-aef90339d241-kube-api-access-nprk9\") pod \"429f70ee-9170-474d-92be-aef90339d241\" (UID: \"429f70ee-9170-474d-92be-aef90339d241\") " Oct 30 01:26:38.848016 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.847984 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-metrics-certs" pod "429f70ee-9170-474d-92be-aef90339d241" (UID: "429f70ee-9170-474d-92be-aef90339d241"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:38.848156 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.848053 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429f70ee-9170-474d-92be-aef90339d241-kube-api-access-nprk9" pod "429f70ee-9170-474d-92be-aef90339d241" (UID: "429f70ee-9170-474d-92be-aef90339d241"). InnerVolumeSpecName "kube-api-access-nprk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:38.848243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.848226 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-webhook-cert" pod "429f70ee-9170-474d-92be-aef90339d241" (UID: "429f70ee-9170-474d-92be-aef90339d241"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:38.940215 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.940173 4927 generic.go:356] "Generic (PLEG): container finished" podID="429f70ee-9170-474d-92be-aef90339d241" containerID="c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c" exitCode=0 Oct 30 01:26:38.940513 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.940234 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-hlvbv" Oct 30 01:26:38.940710 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.940230 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-hlvbv" event={"ID":"429f70ee-9170-474d-92be-aef90339d241","Type":"ContainerDied","Data":"c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c"} Oct 30 01:26:38.940710 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.940649 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-hlvbv" event={"ID":"429f70ee-9170-474d-92be-aef90339d241","Type":"ContainerDied","Data":"fe0050ae31d3aa2373f1447277bf662cbd716a0a69771f493a1d563113568caa"} Oct 30 01:26:38.940710 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.940684 4927 scope.go:122] "RemoveContainer" containerID="c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c" Oct 30 01:26:38.946987 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.946963 4927 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-webhook-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.947086 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.946991 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprk9\" (UniqueName: \"kubernetes.io/projected/429f70ee-9170-474d-92be-aef90339d241-kube-api-access-nprk9\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.947086 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.947007 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429f70ee-9170-474d-92be-aef90339d241-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.955269 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.955246 4927 scope.go:122] "RemoveContainer" containerID="c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c" Oct 30 01:26:38.955658 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:26:38.955603 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c\": container with ID starting with c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c not found: ID does not exist" containerID="c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c" Oct 30 01:26:38.955778 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.955690 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c"} err="failed to get container status \"c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c\": rpc error: code = NotFound desc = could not find container \"c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c\": container with ID starting with c3ce2d4d8f354dfaa308ab1a67d9dcc031b9729538f07c4b8f3a211d565a525c not found: ID does not exist" Oct 30 01:26:38.992752 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.992668 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f6db8855d-hlvbv"] Oct 30 01:26:38.998626 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:38.998600 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f6db8855d-hlvbv"] Oct 30 01:26:39.569559 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:39.569528 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429f70ee-9170-474d-92be-aef90339d241" path="/var/lib/kubelet/pods/429f70ee-9170-474d-92be-aef90339d241/volumes" Oct 30 01:26:40.387747 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.387712 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f6db8855d-m4z7t"] Oct 30 01:26:40.388137 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388123 4927 cpu_manager.go:414] "containerMap: removing container" podUID="429f70ee-9170-474d-92be-aef90339d241" containerName="manager" Oct 30 01:26:40.388192 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388140 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429f70ee-9170-474d-92be-aef90339d241" containerName="manager" Oct 30 01:26:40.388192 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388178 4927 cpu_manager.go:414] "containerMap: removing container" podUID="3540ce90-64ec-436a-9c82-21dd2cc2901a" containerName="payload-pre-processing" Oct 30 01:26:40.388192 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388184 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3540ce90-64ec-436a-9c82-21dd2cc2901a" containerName="payload-pre-processing" Oct 30 01:26:40.388295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388198 4927 cpu_manager.go:414] "containerMap: removing container" podUID="74d6a3b2-8f58-435c-9591-a6d5db1069ce" containerName="payload-processing" Oct 30 01:26:40.388295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388203 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d6a3b2-8f58-435c-9591-a6d5db1069ce" containerName="payload-processing" Oct 30 01:26:40.388295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388212 4927 cpu_manager.go:414] "containerMap: removing container" podUID="424cef29-a2d3-411d-aab4-a379e0e740a5" containerName="revoke-keys" Oct 30 01:26:40.388295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388219 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424cef29-a2d3-411d-aab4-a379e0e740a5" containerName="revoke-keys" Oct 30 01:26:40.388295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388237 4927 cpu_manager.go:414] "containerMap: removing container" podUID="c4670c74-4555-438a-bdb2-b8d9eafb1a19" containerName="payload-processing" Oct 30 01:26:40.388295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388242 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4670c74-4555-438a-bdb2-b8d9eafb1a19" containerName="payload-processing" Oct 30 01:26:40.388489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388315 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="3540ce90-64ec-436a-9c82-21dd2cc2901a" containerName="payload-pre-processing" Oct 30 01:26:40.388489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388325 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d6a3b2-8f58-435c-9591-a6d5db1069ce" containerName="payload-processing" Oct 30 01:26:40.388489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388332 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="424cef29-a2d3-411d-aab4-a379e0e740a5" containerName="revoke-keys" Oct 30 01:26:40.388489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388339 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4670c74-4555-438a-bdb2-b8d9eafb1a19" containerName="payload-processing" Oct 30 01:26:40.388489 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.388346 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="429f70ee-9170-474d-92be-aef90339d241" containerName="manager" Oct 30 01:26:40.393651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.393630 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.421678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.421643 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 01:26:40.421678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.421661 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 01:26:40.421863 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.421677 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5nfwr\"" Oct 30 01:26:40.425756 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.425731 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f6db8855d-m4z7t"] Oct 30 01:26:40.467109 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.467087 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 01:26:40.563490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.563462 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-webhook-cert\") pod \"maas-controller-f6db8855d-m4z7t\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.563676 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.563502 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfhq5\" (UniqueName: \"kubernetes.io/projected/a4c93fe7-5776-41e1-a016-bfb4937c43b3-kube-api-access-sfhq5\") pod \"maas-controller-f6db8855d-m4z7t\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.563676 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.563523 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-metrics-certs\") pod \"maas-controller-f6db8855d-m4z7t\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.664757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.664669 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfhq5\" (UniqueName: \"kubernetes.io/projected/a4c93fe7-5776-41e1-a016-bfb4937c43b3-kube-api-access-sfhq5\") pod \"maas-controller-f6db8855d-m4z7t\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.664757 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.664711 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-metrics-certs\") pod \"maas-controller-f6db8855d-m4z7t\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.665274 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.664850 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-webhook-cert\") pod \"maas-controller-f6db8855d-m4z7t\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.667377 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.667352 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-metrics-certs\") pod \"maas-controller-f6db8855d-m4z7t\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.667498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.667355 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-webhook-cert\") pod \"maas-controller-f6db8855d-m4z7t\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.687253 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.687227 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfhq5\" (UniqueName: \"kubernetes.io/projected/a4c93fe7-5776-41e1-a016-bfb4937c43b3-kube-api-access-sfhq5\") pod \"maas-controller-f6db8855d-m4z7t\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:40.707064 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:40.707034 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:41.044448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:41.044232 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f6db8855d-m4z7t"] Oct 30 01:26:41.047023 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:26:41.046992 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c93fe7_5776_41e1_a016_bfb4937c43b3.slice/crio-6fd59fb7c4304cd692bb5ca3ef5f11890a34a65745871c1a101017ffcc254a64 WatchSource:0}: Error finding container 6fd59fb7c4304cd692bb5ca3ef5f11890a34a65745871c1a101017ffcc254a64: Status 404 returned error can't find the container with id 6fd59fb7c4304cd692bb5ca3ef5f11890a34a65745871c1a101017ffcc254a64 Oct 30 01:26:41.048349 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:41.048329 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:26:41.958800 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:41.958750 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-m4z7t" event={"ID":"a4c93fe7-5776-41e1-a016-bfb4937c43b3","Type":"ContainerStarted","Data":"7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61"} Oct 30 01:26:41.959653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:41.958823 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-m4z7t" event={"ID":"a4c93fe7-5776-41e1-a016-bfb4937c43b3","Type":"ContainerStarted","Data":"6fd59fb7c4304cd692bb5ca3ef5f11890a34a65745871c1a101017ffcc254a64"} Oct 30 01:26:41.959653 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:41.958926 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:42.001181 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:42.001122 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f6db8855d-m4z7t" podStartSLOduration=1.58559259 podStartE2EDuration="2.001106448s" totalImagesPullingTime="415.513858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:41.998839681 +0000 UTC m=+3140.570765903" watchObservedRunningTime="2026-10-30 01:26:42.001106448 +0000 UTC m=+3140.573032669" Oct 30 01:26:52.966686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:52.966650 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:26:57.783017 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.782975 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj"] Oct 30 01:26:57.786824 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.786792 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.815245 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.815210 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj"] Oct 30 01:26:57.830154 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.830116 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.830335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.830172 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.830335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.830212 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpsg\" (UniqueName: \"kubernetes.io/projected/4e5a86cc-5600-4e82-adfe-a60c090ac69e-kube-api-access-4gpsg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.830335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.830281 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.830335 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.830326 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.830635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.830530 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.830728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.830653 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.830799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.830779 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.830862 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.830810 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.931500 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.931463 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.931763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.931526 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.931763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.931566 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.931763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.931623 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.931763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.931666 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.931763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.931688 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.931763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.931723 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.931763 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.931754 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.932252 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.931799 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpsg\" (UniqueName: \"kubernetes.io/projected/4e5a86cc-5600-4e82-adfe-a60c090ac69e-kube-api-access-4gpsg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.932252 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.932174 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.932252 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.932204 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.932383 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.932248 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.932426 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.932403 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.932468 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.932431 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.934021 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.934003 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.934273 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.934255 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.953651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.953621 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5a86cc-5600-4e82-adfe-a60c090ac69e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:57.953651 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:57.953640 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpsg\" (UniqueName: \"kubernetes.io/projected/4e5a86cc-5600-4e82-adfe-a60c090ac69e-kube-api-access-4gpsg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj\" (UID: \"4e5a86cc-5600-4e82-adfe-a60c090ac69e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:58.101073 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:58.100975 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:58.449799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:58.449549 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj"] Oct 30 01:26:58.452223 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:26:58.452196 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5a86cc_5600_4e82_adfe_a60c090ac69e.slice/crio-fcae24b16c9df9a1d1c0861816e466240da958b99824655c34bca67fa1b92a69 WatchSource:0}: Error finding container fcae24b16c9df9a1d1c0861816e466240da958b99824655c34bca67fa1b92a69: Status 404 returned error can't find the container with id fcae24b16c9df9a1d1c0861816e466240da958b99824655c34bca67fa1b92a69 Oct 30 01:26:58.454446 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:58.454411 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:26:58.454553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:58.454494 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:26:58.454553 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:58.454538 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:26:59.056111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:59.056073 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" event={"ID":"4e5a86cc-5600-4e82-adfe-a60c090ac69e","Type":"ContainerStarted","Data":"88ba4a0a8e6549c8923d0eb52f123582adba6a2a88c9a517002a40700b6a535f"} Oct 30 01:26:59.056512 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:59.056128 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" event={"ID":"4e5a86cc-5600-4e82-adfe-a60c090ac69e","Type":"ContainerStarted","Data":"fcae24b16c9df9a1d1c0861816e466240da958b99824655c34bca67fa1b92a69"} Oct 30 01:26:59.101869 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:59.101823 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:26:59.103498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:59.103461 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:26:59.103661 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:26:59.103530 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 01:27:00.102380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:00.102338 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:27:00.102952 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:00.102403 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 01:27:01.101874 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:01.101836 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:27:01.102109 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:01.101901 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 01:27:02.102231 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:02.102174 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:27:02.102818 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:02.102247 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 01:27:03.101627 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:03.101557 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:27:03.101821 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:03.101649 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 01:27:04.102172 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:04.102132 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:27:04.102623 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:04.102196 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 01:27:04.442964 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:27:04.442930 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:27:05.102263 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:05.102223 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:27:05.102749 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:05.102294 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 01:27:07.102415 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:07.102377 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:27:07.102861 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:07.102436 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:27:08.143765 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:08.143729 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:27:08.144197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:08.143803 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:27:08.720619 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:08.720553 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:27:08.720922 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:08.720897 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:27:08.721675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:08.721657 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" Oct 30 01:27:08.777083 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:08.777034 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podStartSLOduration=11.777021423 podStartE2EDuration="11.777021423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:59.125938491 +0000 UTC m=+3157.697864714" watchObservedRunningTime="2026-10-30 01:27:08.777021423 +0000 UTC m=+3167.348947644" Oct 30 01:27:12.825649 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:12.825608 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9"] Oct 30 01:27:12.829793 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:12.829762 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:12.862704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:12.862677 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9"] Oct 30 01:27:13.001675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.001637 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.001870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.001736 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a414ab-cf1f-479f-b762-e231c92450a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.001870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.001772 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a414ab-cf1f-479f-b762-e231c92450a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.001870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.001842 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.001997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.001904 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqz8\" (UniqueName: \"kubernetes.io/projected/52a414ab-cf1f-479f-b762-e231c92450a1-kube-api-access-7rqz8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.001997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.001929 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.001997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.001952 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.001997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.001985 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a414ab-cf1f-479f-b762-e231c92450a1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.002151 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.002023 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.102954 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103004 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a414ab-cf1f-479f-b762-e231c92450a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103032 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103035 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a414ab-cf1f-479f-b762-e231c92450a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103051 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103084 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqz8\" (UniqueName: \"kubernetes.io/projected/52a414ab-cf1f-479f-b762-e231c92450a1-kube-api-access-7rqz8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103106 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103133 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103186 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a414ab-cf1f-479f-b762-e231c92450a1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103332 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103231 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103459 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103514 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103635 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103660 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.103919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.103898 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a414ab-cf1f-479f-b762-e231c92450a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.105555 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.105532 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a414ab-cf1f-479f-b762-e231c92450a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.105685 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.105635 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a414ab-cf1f-479f-b762-e231c92450a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.125362 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.125329 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a414ab-cf1f-479f-b762-e231c92450a1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.125542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.125522 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqz8\" (UniqueName: \"kubernetes.io/projected/52a414ab-cf1f-479f-b762-e231c92450a1-kube-api-access-7rqz8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-885k9\" (UID: \"52a414ab-cf1f-479f-b762-e231c92450a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.144342 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.144318 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:13.414579 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.414534 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f6db8855d-m4z7t"] Oct 30 01:27:13.414770 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.414749 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-f6db8855d-m4z7t" podUID="a4c93fe7-5776-41e1-a016-bfb4937c43b3" containerName="manager" containerID="cri-o://7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61" gracePeriod=30 Oct 30 01:27:13.474825 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.474800 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9"] Oct 30 01:27:13.500592 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:27:13.500548 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a414ab_cf1f_479f_b762_e231c92450a1.slice/crio-2ad7dd3b8a57e6135eb7c512574c969ab857fa5ce62402afb716fc549cdb9913 WatchSource:0}: Error finding container 2ad7dd3b8a57e6135eb7c512574c969ab857fa5ce62402afb716fc549cdb9913: Status 404 returned error can't find the container with id 2ad7dd3b8a57e6135eb7c512574c969ab857fa5ce62402afb716fc549cdb9913 Oct 30 01:27:13.520870 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.520835 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:27:13.520982 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.520903 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:27:13.520982 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.520936 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:27:13.668929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.668863 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:27:13.809268 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.809230 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-webhook-cert\") pod \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " Oct 30 01:27:13.809440 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.809324 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-metrics-certs\") pod \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " Oct 30 01:27:13.809490 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.809463 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c93fe7-5776-41e1-a016-bfb4937c43b3-kube-api-access-sfhq5\" (UniqueName: \"kubernetes.io/projected/a4c93fe7-5776-41e1-a016-bfb4937c43b3-kube-api-access-sfhq5\") pod \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\" (UID: \"a4c93fe7-5776-41e1-a016-bfb4937c43b3\") " Oct 30 01:27:13.811493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.811464 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-metrics-certs" pod "a4c93fe7-5776-41e1-a016-bfb4937c43b3" (UID: "a4c93fe7-5776-41e1-a016-bfb4937c43b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:13.811493 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.811484 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-webhook-cert" pod "a4c93fe7-5776-41e1-a016-bfb4937c43b3" (UID: "a4c93fe7-5776-41e1-a016-bfb4937c43b3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:13.811710 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.811496 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c93fe7-5776-41e1-a016-bfb4937c43b3-kube-api-access-sfhq5" pod "a4c93fe7-5776-41e1-a016-bfb4937c43b3" (UID: "a4c93fe7-5776-41e1-a016-bfb4937c43b3"). InnerVolumeSpecName "kube-api-access-sfhq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:13.910635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.910556 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:27:13.910635 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.910642 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfhq5\" (UniqueName: \"kubernetes.io/projected/a4c93fe7-5776-41e1-a016-bfb4937c43b3-kube-api-access-sfhq5\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:27:13.911313 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:13.910663 4927 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4c93fe7-5776-41e1-a016-bfb4937c43b3-webhook-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:27:14.163642 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.163602 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" event={"ID":"52a414ab-cf1f-479f-b762-e231c92450a1","Type":"ContainerStarted","Data":"0637b7dee1273eb82ef8bbdd3998087389bd68eee85effb0a095875f219db019"} Oct 30 01:27:14.163833 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.163668 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" event={"ID":"52a414ab-cf1f-479f-b762-e231c92450a1","Type":"ContainerStarted","Data":"2ad7dd3b8a57e6135eb7c512574c969ab857fa5ce62402afb716fc549cdb9913"} Oct 30 01:27:14.164957 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.164926 4927 generic.go:356] "Generic (PLEG): container finished" podID="a4c93fe7-5776-41e1-a016-bfb4937c43b3" containerID="7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61" exitCode=0 Oct 30 01:27:14.165062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.164958 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-m4z7t" event={"ID":"a4c93fe7-5776-41e1-a016-bfb4937c43b3","Type":"ContainerDied","Data":"7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61"} Oct 30 01:27:14.165062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.164964 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-m4z7t" Oct 30 01:27:14.165062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.164987 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-m4z7t" event={"ID":"a4c93fe7-5776-41e1-a016-bfb4937c43b3","Type":"ContainerDied","Data":"6fd59fb7c4304cd692bb5ca3ef5f11890a34a65745871c1a101017ffcc254a64"} Oct 30 01:27:14.165062 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.165009 4927 scope.go:122] "RemoveContainer" containerID="7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61" Oct 30 01:27:14.202678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.202647 4927 scope.go:122] "RemoveContainer" containerID="7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61" Oct 30 01:27:14.203314 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:27:14.203274 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61\": container with ID starting with 7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61 not found: ID does not exist" containerID="7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61" Oct 30 01:27:14.203439 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.203362 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61"} err="failed to get container status \"7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61\": rpc error: code = NotFound desc = could not find container \"7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61\": container with ID starting with 7e5ed9ad0a482972d4a6fc23ee0b3c4d96a256e55e9d46acbd5f2bfb4b034a61 not found: ID does not exist" Oct 30 01:27:14.228509 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.228456 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" podStartSLOduration=2.228444309 podStartE2EDuration="2.228444309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:14.226228001 +0000 UTC m=+3172.798154223" watchObservedRunningTime="2026-10-30 01:27:14.228444309 +0000 UTC m=+3172.800370531" Oct 30 01:27:14.262452 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.262421 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f6db8855d-m4z7t"] Oct 30 01:27:14.266892 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:14.266868 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f6db8855d-m4z7t"] Oct 30 01:27:15.145496 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.145456 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:15.147202 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.147172 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-885k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 01:27:15.147352 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.147225 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" podUID="52a414ab-cf1f-479f-b762-e231c92450a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 01:27:15.472143 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.472063 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f6db8855d-587kv"] Oct 30 01:27:15.472477 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.472464 4927 cpu_manager.go:414] "containerMap: removing container" podUID="a4c93fe7-5776-41e1-a016-bfb4937c43b3" containerName="manager" Oct 30 01:27:15.472522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.472478 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c93fe7-5776-41e1-a016-bfb4937c43b3" containerName="manager" Oct 30 01:27:15.472596 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.472564 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c93fe7-5776-41e1-a016-bfb4937c43b3" containerName="manager" Oct 30 01:27:15.499145 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.499116 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.505285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.505258 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f6db8855d-587kv"] Oct 30 01:27:15.513407 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.513379 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 01:27:15.513554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.513487 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5nfwr\"" Oct 30 01:27:15.524659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.524637 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 01:27:15.524659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.524641 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 01:27:15.570561 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.570526 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c93fe7-5776-41e1-a016-bfb4937c43b3" path="/var/lib/kubelet/pods/a4c93fe7-5776-41e1-a016-bfb4937c43b3/volumes" Oct 30 01:27:15.628685 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.628650 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93c060f9-2b8d-4c98-a339-cde854bb5784-webhook-cert\") pod \"maas-controller-f6db8855d-587kv\" (UID: \"93c060f9-2b8d-4c98-a339-cde854bb5784\") " pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.628885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.628777 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c060f9-2b8d-4c98-a339-cde854bb5784-metrics-certs\") pod \"maas-controller-f6db8855d-587kv\" (UID: \"93c060f9-2b8d-4c98-a339-cde854bb5784\") " pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.628885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.628823 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns7lf\" (UniqueName: \"kubernetes.io/projected/93c060f9-2b8d-4c98-a339-cde854bb5784-kube-api-access-ns7lf\") pod \"maas-controller-f6db8855d-587kv\" (UID: \"93c060f9-2b8d-4c98-a339-cde854bb5784\") " pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.730037 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.729942 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c060f9-2b8d-4c98-a339-cde854bb5784-metrics-certs\") pod \"maas-controller-f6db8855d-587kv\" (UID: \"93c060f9-2b8d-4c98-a339-cde854bb5784\") " pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.730231 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.730038 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns7lf\" (UniqueName: \"kubernetes.io/projected/93c060f9-2b8d-4c98-a339-cde854bb5784-kube-api-access-ns7lf\") pod \"maas-controller-f6db8855d-587kv\" (UID: \"93c060f9-2b8d-4c98-a339-cde854bb5784\") " pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.730231 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.730108 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93c060f9-2b8d-4c98-a339-cde854bb5784-webhook-cert\") pod \"maas-controller-f6db8855d-587kv\" (UID: \"93c060f9-2b8d-4c98-a339-cde854bb5784\") " pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.732833 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.732803 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c060f9-2b8d-4c98-a339-cde854bb5784-metrics-certs\") pod \"maas-controller-f6db8855d-587kv\" (UID: \"93c060f9-2b8d-4c98-a339-cde854bb5784\") " pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.732950 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.732803 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93c060f9-2b8d-4c98-a339-cde854bb5784-webhook-cert\") pod \"maas-controller-f6db8855d-587kv\" (UID: \"93c060f9-2b8d-4c98-a339-cde854bb5784\") " pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.752448 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.752419 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns7lf\" (UniqueName: \"kubernetes.io/projected/93c060f9-2b8d-4c98-a339-cde854bb5784-kube-api-access-ns7lf\") pod \"maas-controller-f6db8855d-587kv\" (UID: \"93c060f9-2b8d-4c98-a339-cde854bb5784\") " pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:15.814764 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:15.814733 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:16.125749 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:16.125682 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f6db8855d-587kv"] Oct 30 01:27:16.128171 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:27:16.128142 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c060f9_2b8d_4c98_a339_cde854bb5784.slice/crio-c3c7b305ad65562cd38637885551046760169c8f956f215a172989f43a3abe22 WatchSource:0}: Error finding container c3c7b305ad65562cd38637885551046760169c8f956f215a172989f43a3abe22: Status 404 returned error can't find the container with id c3c7b305ad65562cd38637885551046760169c8f956f215a172989f43a3abe22 Oct 30 01:27:16.145006 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:16.144979 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-885k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 01:27:16.145122 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:16.145033 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" podUID="52a414ab-cf1f-479f-b762-e231c92450a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 01:27:16.204930 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:16.204875 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-587kv" event={"ID":"93c060f9-2b8d-4c98-a339-cde854bb5784","Type":"ContainerStarted","Data":"c3c7b305ad65562cd38637885551046760169c8f956f215a172989f43a3abe22"} Oct 30 01:27:17.144883 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:17.144845 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-885k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 01:27:17.145076 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:17.144905 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" podUID="52a414ab-cf1f-479f-b762-e231c92450a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 01:27:17.213646 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:17.213598 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6db8855d-587kv" event={"ID":"93c060f9-2b8d-4c98-a339-cde854bb5784","Type":"ContainerStarted","Data":"24b366ba1f19e4901848e9e25b828e0820fca21717a20f5ac40085c6cdb2b97f"} Oct 30 01:27:17.214115 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:17.213672 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:17.257418 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:17.257367 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f6db8855d-587kv" podStartSLOduration=1.8862727590000001 podStartE2EDuration="2.257352675s" totalImagesPullingTime="371.079916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:17.254689063 +0000 UTC m=+3175.826615286" watchObservedRunningTime="2026-10-30 01:27:17.257352675 +0000 UTC m=+3175.829278943" Oct 30 01:27:18.145389 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:18.145353 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-885k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 01:27:18.145613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:18.145405 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" podUID="52a414ab-cf1f-479f-b762-e231c92450a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 01:27:19.145525 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:19.145482 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-885k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 01:27:19.145995 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:19.145537 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" podUID="52a414ab-cf1f-479f-b762-e231c92450a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 01:27:20.144806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:20.144765 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-885k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 01:27:20.144969 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:20.144837 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" podUID="52a414ab-cf1f-479f-b762-e231c92450a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 01:27:21.779199 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:21.779162 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:21.780023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:21.779998 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:21.780551 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:21.780533 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" Oct 30 01:27:28.221968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:28.221936 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f6db8855d-587kv" Oct 30 01:27:34.441495 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:27:34.441415 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:27:46.715595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:46.714673 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:27:46.715595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:27:46.714746 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podUID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:28:04.442306 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:28:04.442266 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:28:07.846700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:07.846485 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-885k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:28:07.846700 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:07.846589 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" podUID="52a414ab-cf1f-479f-b762-e231c92450a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:28:13.011295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.011255 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz"] Oct 30 01:28:13.015328 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.015310 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.047140 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.047113 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz"] Oct 30 01:28:13.115852 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.115817 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d429408-e1f3-4a7a-8047-26f5861af69d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.116025 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.115996 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.116096 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.116058 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.116191 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.116151 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.116247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.116188 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.116247 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.116214 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.116322 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.116304 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.116366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.116341 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5wk8\" (UniqueName: \"kubernetes.io/projected/8d429408-e1f3-4a7a-8047-26f5861af69d-kube-api-access-c5wk8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.116366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.116359 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.217542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.217501 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.217542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.217550 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.217843 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.217605 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.217843 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.217626 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.217843 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.217642 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.217843 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.217726 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.218074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.217879 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wk8\" (UniqueName: \"kubernetes.io/projected/8d429408-e1f3-4a7a-8047-26f5861af69d-kube-api-access-c5wk8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.218074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.217922 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.218074 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.217954 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d429408-e1f3-4a7a-8047-26f5861af69d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.218238 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.218079 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.218238 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.218120 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.218354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.218268 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.218354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.218334 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.218658 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.218638 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d429408-e1f3-4a7a-8047-26f5861af69d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.220006 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.219982 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.220224 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.220209 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.240560 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.240536 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d429408-e1f3-4a7a-8047-26f5861af69d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.240684 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.240601 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wk8\" (UniqueName: \"kubernetes.io/projected/8d429408-e1f3-4a7a-8047-26f5861af69d-kube-api-access-c5wk8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4fglz\" (UID: \"8d429408-e1f3-4a7a-8047-26f5861af69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.329028 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.328934 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:13.659378 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.659345 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz"] Oct 30 01:28:13.662172 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:28:13.662143 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d429408_e1f3_4a7a_8047_26f5861af69d.slice/crio-6b25ce4bc3d701e5a022173419268a82d6f565127f83537278e19ec622309e30 WatchSource:0}: Error finding container 6b25ce4bc3d701e5a022173419268a82d6f565127f83537278e19ec622309e30: Status 404 returned error can't find the container with id 6b25ce4bc3d701e5a022173419268a82d6f565127f83537278e19ec622309e30 Oct 30 01:28:13.664372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.664338 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:28:13.664447 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.664404 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:28:13.664447 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:13.664441 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:28:14.577603 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:14.575606 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" event={"ID":"8d429408-e1f3-4a7a-8047-26f5861af69d","Type":"ContainerStarted","Data":"641ad5c93ffae8bf49044c81674d8a1da1752ee79b5eac3cb8254dcf08ca6fc7"} Oct 30 01:28:14.577603 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:14.575674 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" event={"ID":"8d429408-e1f3-4a7a-8047-26f5861af69d","Type":"ContainerStarted","Data":"6b25ce4bc3d701e5a022173419268a82d6f565127f83537278e19ec622309e30"} Oct 30 01:28:14.632483 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:14.632425 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" podStartSLOduration=2.632410013 podStartE2EDuration="2.632410013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:14.619377801 +0000 UTC m=+3233.191304024" watchObservedRunningTime="2026-10-30 01:28:14.632410013 +0000 UTC m=+3233.204336232" Oct 30 01:28:15.329776 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:15.329742 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:15.331461 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:15.331438 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:28:15.331534 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:15.331484 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" podUID="8d429408-e1f3-4a7a-8047-26f5861af69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:28:16.329951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:16.329908 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:28:16.330372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:16.329965 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" podUID="8d429408-e1f3-4a7a-8047-26f5861af69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:28:17.329305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:17.329269 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:28:17.329511 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:17.329324 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" podUID="8d429408-e1f3-4a7a-8047-26f5861af69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:28:18.330141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:18.330098 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:28:18.330558 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:18.330160 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" podUID="8d429408-e1f3-4a7a-8047-26f5861af69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:28:19.329733 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:19.329693 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:28:19.329936 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:19.329752 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" podUID="8d429408-e1f3-4a7a-8047-26f5861af69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:28:20.329352 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:20.329310 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:28:20.329767 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:20.329395 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" podUID="8d429408-e1f3-4a7a-8047-26f5861af69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 01:28:21.904976 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:21.904941 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:21.905640 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:21.905600 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:21.906235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:28:21.906192 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" Oct 30 01:28:34.441486 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:28:34.441439 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:29:04.462086 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:29:04.461990 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:29:22.630608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:29:22.630563 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:29:22.650701 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:29:22.650677 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:29:34.441926 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:29:34.441883 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:29:37.844721 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:29:37.844671 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-885k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:29:37.845371 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:29:37.845341 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-885k9" podUID="52a414ab-cf1f-479f-b762-e231c92450a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:29:37.958738 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:29:37.958691 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:29:37.960099 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:29:37.960062 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4fglz" podUID="8d429408-e1f3-4a7a-8047-26f5861af69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:30:01.704798 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:30:01.704738 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:30:01.705614 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:30:01.705579 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podUID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:30:04.443386 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:30:04.443342 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:30:34.442397 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:30:34.442310 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:31:04.492621 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:31:04.492541 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:31:14.027662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:14.027628 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn"] Oct 30 01:31:14.038920 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:14.038892 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffee52f-g27dn"] Oct 30 01:31:15.569892 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:15.569854 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424cef29-a2d3-411d-aab4-a379e0e740a5" path="/var/lib/kubelet/pods/424cef29-a2d3-411d-aab4-a379e0e740a5/volumes" Oct 30 01:31:26.217993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:26.217949 4927 scope.go:122] "RemoveContainer" containerID="e774eb7b5cc1de5a104ccc5a3bb6812e7a38507fbeeddde975a15f7f809f72ef" Oct 30 01:31:34.444143 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:31:34.444101 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:31:37.466366 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.466301 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz"] Oct 30 01:31:37.471688 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.471660 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.482116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.482087 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-data\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.482253 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.482129 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e704866-a137-45cd-9aed-3b76635e1409-istio-podinfo\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.482301 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.482249 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-istio-token\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.482357 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.482304 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-certs\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.482410 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.482355 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-socket\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.482498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.482481 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-envoy\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.482539 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.482522 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-credential-socket\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.482605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.482564 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsczn\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-kube-api-access-fsczn\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.482704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.482687 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e704866-a137-45cd-9aed-3b76635e1409-istiod-ca-cert\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.487030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.487006 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ff1bf5-openshift-default-dockercfg-5nrvh\"" Oct 30 01:31:37.501127 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.501101 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz"] Oct 30 01:31:37.583419 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583383 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-envoy\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.583419 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583422 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-credential-socket\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.583748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583447 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsczn\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-kube-api-access-fsczn\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.583748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583470 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e704866-a137-45cd-9aed-3b76635e1409-istiod-ca-cert\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.583748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583526 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-data\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.583748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583545 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e704866-a137-45cd-9aed-3b76635e1409-istio-podinfo\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.583748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583623 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-istio-token\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.583748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583660 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-certs\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.583748 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583704 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-socket\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.584135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.583982 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-data\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.584135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.584005 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-credential-socket\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.584135 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.584089 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-certs\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.584308 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.584141 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-socket\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.584416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.584393 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e704866-a137-45cd-9aed-3b76635e1409-istiod-ca-cert\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.585879 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.585858 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-envoy\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.586424 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.586401 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e704866-a137-45cd-9aed-3b76635e1409-istio-podinfo\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.605699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.605674 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-istio-token\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.605973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.605949 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsczn\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-kube-api-access-fsczn\") pod \"e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:37.786963 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:37.786877 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:38.109728 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:38.109698 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz"] Oct 30 01:31:38.112636 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:31:38.112609 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e704866_a137_45cd_9aed_3b76635e1409.slice/crio-630b5e3a31d8939565b97b22c9001771eea3340f2e810a8853a9d17125ca8cee WatchSource:0}: Error finding container 630b5e3a31d8939565b97b22c9001771eea3340f2e810a8853a9d17125ca8cee: Status 404 returned error can't find the container with id 630b5e3a31d8939565b97b22c9001771eea3340f2e810a8853a9d17125ca8cee Oct 30 01:31:38.115010 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:38.114958 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:31:38.115116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:38.115040 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:31:38.115116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:38.115085 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:31:38.143005 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:38.142970 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" event={"ID":"8e704866-a137-45cd-9aed-3b76635e1409","Type":"ContainerStarted","Data":"630b5e3a31d8939565b97b22c9001771eea3340f2e810a8853a9d17125ca8cee"} Oct 30 01:31:39.150804 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:39.150757 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" event={"ID":"8e704866-a137-45cd-9aed-3b76635e1409","Type":"ContainerStarted","Data":"1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c"} Oct 30 01:31:39.206142 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:39.206086 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" podStartSLOduration=2.206072609 podStartE2EDuration="2.206072609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:39.193146097 +0000 UTC m=+3437.765072318" watchObservedRunningTime="2026-10-30 01:31:39.206072609 +0000 UTC m=+3437.777998831" Oct 30 01:31:39.787269 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:39.787234 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:39.792452 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:39.792426 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:40.156745 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:40.156708 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:40.157708 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:40.157684 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:43.346242 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.346207 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt"] Oct 30 01:31:43.350277 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.350255 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:43.367168 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.367145 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ff1bf5\"" Oct 30 01:31:43.378687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.378655 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4a8a3e9b-dockercfg-mr4vf\"" Oct 30 01:31:43.381030 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.381005 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt"] Oct 30 01:31:43.409500 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.409470 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb"] Oct 30 01:31:43.413167 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.413145 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:43.433148 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.433124 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb"] Oct 30 01:31:43.470243 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.469447 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427427dd-4d81-40cb-8944-117049c023b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb\" (UID: \"427427dd-4d81-40cb-8944-117049c023b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:43.470929 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.470883 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67b9aaf-658e-4715-8f99-7f1b45c95833-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt\" (UID: \"f67b9aaf-658e-4715-8f99-7f1b45c95833\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:43.471298 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.471212 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7st2\" (UniqueName: \"kubernetes.io/projected/427427dd-4d81-40cb-8944-117049c023b4-kube-api-access-w7st2\") pod \"payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb\" (UID: \"427427dd-4d81-40cb-8944-117049c023b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:43.471604 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.471522 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vtf\" (UniqueName: \"kubernetes.io/projected/f67b9aaf-658e-4715-8f99-7f1b45c95833-kube-api-access-d2vtf\") pod \"payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt\" (UID: \"f67b9aaf-658e-4715-8f99-7f1b45c95833\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:43.574600 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.574543 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427427dd-4d81-40cb-8944-117049c023b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb\" (UID: \"427427dd-4d81-40cb-8944-117049c023b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:43.574931 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.574904 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67b9aaf-658e-4715-8f99-7f1b45c95833-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt\" (UID: \"f67b9aaf-658e-4715-8f99-7f1b45c95833\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:43.575129 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.575113 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7st2\" (UniqueName: \"kubernetes.io/projected/427427dd-4d81-40cb-8944-117049c023b4-kube-api-access-w7st2\") pod \"payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb\" (UID: \"427427dd-4d81-40cb-8944-117049c023b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:43.575372 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.575354 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vtf\" (UniqueName: \"kubernetes.io/projected/f67b9aaf-658e-4715-8f99-7f1b45c95833-kube-api-access-d2vtf\") pod \"payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt\" (UID: \"f67b9aaf-658e-4715-8f99-7f1b45c95833\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:43.576707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.576681 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427427dd-4d81-40cb-8944-117049c023b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb\" (UID: \"427427dd-4d81-40cb-8944-117049c023b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:43.577623 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.577599 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67b9aaf-658e-4715-8f99-7f1b45c95833-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt\" (UID: \"f67b9aaf-658e-4715-8f99-7f1b45c95833\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:43.597675 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.597620 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vtf\" (UniqueName: \"kubernetes.io/projected/f67b9aaf-658e-4715-8f99-7f1b45c95833-kube-api-access-d2vtf\") pod \"payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt\" (UID: \"f67b9aaf-658e-4715-8f99-7f1b45c95833\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:43.599662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.599642 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7st2\" (UniqueName: \"kubernetes.io/projected/427427dd-4d81-40cb-8944-117049c023b4-kube-api-access-w7st2\") pod \"payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb\" (UID: \"427427dd-4d81-40cb-8944-117049c023b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:43.664681 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.664638 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:43.727067 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.726982 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:43.967443 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:43.967414 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt"] Oct 30 01:31:43.969610 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:31:43.969551 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67b9aaf_658e_4715_8f99_7f1b45c95833.slice/crio-e6537bd69e6b2631d72e24e152008555ae02670a9a7ce41fc5ab0c8a5f2b8865 WatchSource:0}: Error finding container e6537bd69e6b2631d72e24e152008555ae02670a9a7ce41fc5ab0c8a5f2b8865: Status 404 returned error can't find the container with id e6537bd69e6b2631d72e24e152008555ae02670a9a7ce41fc5ab0c8a5f2b8865 Oct 30 01:31:44.065162 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:44.065136 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb"] Oct 30 01:31:44.066849 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:31:44.066817 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427427dd_4d81_40cb_8944_117049c023b4.slice/crio-451ec820390c082b6d94706a61daae45718fc54f01f32d8652d7944efb34cb24 WatchSource:0}: Error finding container 451ec820390c082b6d94706a61daae45718fc54f01f32d8652d7944efb34cb24: Status 404 returned error can't find the container with id 451ec820390c082b6d94706a61daae45718fc54f01f32d8652d7944efb34cb24 Oct 30 01:31:44.203208 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:44.203158 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" event={"ID":"427427dd-4d81-40cb-8944-117049c023b4","Type":"ContainerStarted","Data":"8f5e8356bf2c6421be38123e11cd43dabf3a00d9cc275e45895206bd8159f60c"} Oct 30 01:31:44.203394 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:44.203225 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" event={"ID":"427427dd-4d81-40cb-8944-117049c023b4","Type":"ContainerStarted","Data":"451ec820390c082b6d94706a61daae45718fc54f01f32d8652d7944efb34cb24"} Oct 30 01:31:44.203451 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:44.203384 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:44.205220 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:44.205194 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" event={"ID":"f67b9aaf-658e-4715-8f99-7f1b45c95833","Type":"ContainerStarted","Data":"b2177326aa33f255d1606284a7a1f043ab699fd063e843bea41c156b9e1ac705"} Oct 30 01:31:44.205682 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:44.205230 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" event={"ID":"f67b9aaf-658e-4715-8f99-7f1b45c95833","Type":"ContainerStarted","Data":"e6537bd69e6b2631d72e24e152008555ae02670a9a7ce41fc5ab0c8a5f2b8865"} Oct 30 01:31:44.205682 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:44.205260 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:44.252395 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:44.252286 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" podStartSLOduration=1.252269138 podStartE2EDuration="1.252269138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:44.24968338 +0000 UTC m=+3442.821609645" watchObservedRunningTime="2026-10-30 01:31:44.252269138 +0000 UTC m=+3442.824195359" Oct 30 01:31:44.290678 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:44.290622 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" podStartSLOduration=1.290608297 podStartE2EDuration="1.290608297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:44.28791691 +0000 UTC m=+3442.859843134" watchObservedRunningTime="2026-10-30 01:31:44.290608297 +0000 UTC m=+3442.862534519" Oct 30 01:31:55.214318 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:55.214283 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:31:55.214854 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:55.214507 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:31:56.198750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:56.198710 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:31:56.198993 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:56.198782 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:31:57.282395 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:57.282358 4927 generic.go:356] "Generic (PLEG): container finished" podID="8e704866-a137-45cd-9aed-3b76635e1409" containerID="1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c" exitCode=0 Oct 30 01:31:57.283015 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:57.282410 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" event={"ID":"8e704866-a137-45cd-9aed-3b76635e1409","Type":"ContainerDied","Data":"1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c"} Oct 30 01:31:57.283015 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:57.282933 4927 scope.go:122] "RemoveContainer" containerID="1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c" Oct 30 01:31:57.284278 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:57.284245 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:31:57.284353 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:57.284319 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:31:57.284400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:57.284351 4927 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:31:58.304951 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:58.304914 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" event={"ID":"8e704866-a137-45cd-9aed-3b76635e1409","Type":"ContainerStarted","Data":"c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f"} Oct 30 01:31:58.788592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:58.787995 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:31:58.788592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:58.788133 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:31:58.788592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:58.788196 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 01:31:59.795595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:59.789980 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:31:59.795595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:31:59.790058 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 01:32:00.798595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:00.797954 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:32:00.798595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:00.798021 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 01:32:01.793597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:01.790968 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:32:01.793597 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:01.791044 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 01:32:02.797660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:02.797419 4927 patch_prober.go:29] interesting pod/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:32:02.797660 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:02.797488 4927 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 01:32:03.898229 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:03.898193 4927 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:32:03.898837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:03.898814 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:32:03.899919 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:03.899889 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:32:04.442739 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:32:04.442696 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:32:26.244210 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:26.244172 4927 scope.go:122] "RemoveContainer" containerID="91d23814acf449cd94de5bb8e4c1332b638c12e0bc22f683ef2d3113904b8f2f" Oct 30 01:32:34.442552 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:32:34.442510 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:32:43.495057 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:43.495022 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt"] Oct 30 01:32:43.528494 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:43.528461 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb"] Oct 30 01:32:48.495864 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:48.495780 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" podUID="f67b9aaf-658e-4715-8f99-7f1b45c95833" containerName="payload-processing" containerID="cri-o://b2177326aa33f255d1606284a7a1f043ab699fd063e843bea41c156b9e1ac705" gracePeriod=25 Oct 30 01:32:48.529676 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:48.529630 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" podUID="427427dd-4d81-40cb-8944-117049c023b4" containerName="payload-pre-processing" containerID="cri-o://8f5e8356bf2c6421be38123e11cd43dabf3a00d9cc275e45895206bd8159f60c" gracePeriod=25 Oct 30 01:32:53.907305 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:53.907252 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz"] Oct 30 01:32:53.907937 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:53.907497 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" containerID="cri-o://c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f" gracePeriod=30 Oct 30 01:32:55.222625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:55.212743 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" podUID="427427dd-4d81-40cb-8944-117049c023b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:32:55.222625 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:55.213534 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" podUID="f67b9aaf-658e-4715-8f99-7f1b45c95833" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:32:59.180238 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.180174 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:32:59.264134 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.264094 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-istio-token\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-istio-token\") pod \"8e704866-a137-45cd-9aed-3b76635e1409\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " Oct 30 01:32:59.264346 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.264154 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-envoy\") pod \"8e704866-a137-45cd-9aed-3b76635e1409\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " Oct 30 01:32:59.264346 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.264268 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e704866-a137-45cd-9aed-3b76635e1409-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e704866-a137-45cd-9aed-3b76635e1409-istio-podinfo\") pod \"8e704866-a137-45cd-9aed-3b76635e1409\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " Oct 30 01:32:59.264492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.264374 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-data\") pod \"8e704866-a137-45cd-9aed-3b76635e1409\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " Oct 30 01:32:59.264492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.264405 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-socket\") pod \"8e704866-a137-45cd-9aed-3b76635e1409\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " Oct 30 01:32:59.264492 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.264440 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e704866-a137-45cd-9aed-3b76635e1409-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e704866-a137-45cd-9aed-3b76635e1409-istiod-ca-cert\") pod \"8e704866-a137-45cd-9aed-3b76635e1409\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " Oct 30 01:32:59.264686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.264507 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-kube-api-access-fsczn\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-kube-api-access-fsczn\") pod \"8e704866-a137-45cd-9aed-3b76635e1409\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " Oct 30 01:32:59.264686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.264617 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-certs\") pod \"8e704866-a137-45cd-9aed-3b76635e1409\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " Oct 30 01:32:59.264686 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.264652 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-credential-socket\") pod \"8e704866-a137-45cd-9aed-3b76635e1409\" (UID: \"8e704866-a137-45cd-9aed-3b76635e1409\") " Oct 30 01:32:59.265737 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.265005 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-socket" pod "8e704866-a137-45cd-9aed-3b76635e1409" (UID: "8e704866-a137-45cd-9aed-3b76635e1409"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:59.265737 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.265133 4927 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.265737 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.265243 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-credential-socket" pod "8e704866-a137-45cd-9aed-3b76635e1409" (UID: "8e704866-a137-45cd-9aed-3b76635e1409"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:59.265737 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.265345 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e704866-a137-45cd-9aed-3b76635e1409-istiod-ca-cert" pod "8e704866-a137-45cd-9aed-3b76635e1409" (UID: "8e704866-a137-45cd-9aed-3b76635e1409"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:59.265737 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.265655 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-certs" pod "8e704866-a137-45cd-9aed-3b76635e1409" (UID: "8e704866-a137-45cd-9aed-3b76635e1409"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:59.265737 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.265711 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-data" pod "8e704866-a137-45cd-9aed-3b76635e1409" (UID: "8e704866-a137-45cd-9aed-3b76635e1409"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:59.266632 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.266602 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-istio-token" pod "8e704866-a137-45cd-9aed-3b76635e1409" (UID: "8e704866-a137-45cd-9aed-3b76635e1409"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:59.266860 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.266841 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-envoy" pod "8e704866-a137-45cd-9aed-3b76635e1409" (UID: "8e704866-a137-45cd-9aed-3b76635e1409"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:59.267104 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.267078 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e704866-a137-45cd-9aed-3b76635e1409-istio-podinfo" pod "8e704866-a137-45cd-9aed-3b76635e1409" (UID: "8e704866-a137-45cd-9aed-3b76635e1409"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:59.267380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.267350 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-kube-api-access-fsczn" pod "8e704866-a137-45cd-9aed-3b76635e1409" (UID: "8e704866-a137-45cd-9aed-3b76635e1409"). InnerVolumeSpecName "kube-api-access-fsczn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:59.366542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.366502 4927 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-workload-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.366542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.366537 4927 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-credential-socket\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.366542 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.366547 4927 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-istio-token\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.366847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.366561 4927 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-envoy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.366847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.366600 4927 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e704866-a137-45cd-9aed-3b76635e1409-istio-podinfo\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.366847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.366609 4927 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e704866-a137-45cd-9aed-3b76635e1409-istio-data\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.366847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.366617 4927 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e704866-a137-45cd-9aed-3b76635e1409-istiod-ca-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.366847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.366625 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsczn\" (UniqueName: \"kubernetes.io/projected/8e704866-a137-45cd-9aed-3b76635e1409-kube-api-access-fsczn\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.675750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.675714 4927 generic.go:356] "Generic (PLEG): container finished" podID="8e704866-a137-45cd-9aed-3b76635e1409" containerID="c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f" exitCode=0 Oct 30 01:32:59.675750 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.675756 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" event={"ID":"8e704866-a137-45cd-9aed-3b76635e1409","Type":"ContainerDied","Data":"c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f"} Oct 30 01:32:59.675980 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.675771 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" Oct 30 01:32:59.675980 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.675806 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz" event={"ID":"8e704866-a137-45cd-9aed-3b76635e1409","Type":"ContainerDied","Data":"630b5e3a31d8939565b97b22c9001771eea3340f2e810a8853a9d17125ca8cee"} Oct 30 01:32:59.675980 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.675831 4927 scope.go:122] "RemoveContainer" containerID="c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f" Oct 30 01:32:59.688362 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.688342 4927 scope.go:122] "RemoveContainer" containerID="1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c" Oct 30 01:32:59.697313 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.697295 4927 scope.go:122] "RemoveContainer" containerID="c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f" Oct 30 01:32:59.697565 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:32:59.697542 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f\": container with ID starting with c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f not found: ID does not exist" containerID="c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f" Oct 30 01:32:59.697647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.697586 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f"} err="failed to get container status \"c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f\": rpc error: code = NotFound desc = could not find container \"c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f\": container with ID starting with c076fa4d458957fb59dc571fb3ac61bcb9d9095b14178928ebf5baa377d5ef5f not found: ID does not exist" Oct 30 01:32:59.697647 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.697609 4927 scope.go:122] "RemoveContainer" containerID="1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c" Oct 30 01:32:59.697849 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:32:59.697834 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c\": container with ID starting with 1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c not found: ID does not exist" containerID="1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c" Oct 30 01:32:59.697895 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.697853 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c"} err="failed to get container status \"1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c\": rpc error: code = NotFound desc = could not find container \"1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c\": container with ID starting with 1022ab5f5eae58c91b940b74438007c672f65f2643552e2df72c325d8fbe906c not found: ID does not exist" Oct 30 01:32:59.726390 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.726360 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz"] Oct 30 01:32:59.737659 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:32:59.737628 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ff1bf5-openshift-default-b756b9997-vwhkz"] Oct 30 01:33:01.570228 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:01.570193 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e704866-a137-45cd-9aed-3b76635e1409" path="/var/lib/kubelet/pods/8e704866-a137-45cd-9aed-3b76635e1409/volumes" Oct 30 01:33:04.441364 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:33:04.441323 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:33:05.212423 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:05.212382 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" podUID="f67b9aaf-658e-4715-8f99-7f1b45c95833" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:33:05.212629 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:05.212593 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" podUID="427427dd-4d81-40cb-8944-117049c023b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:33:08.781303 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:08.781027 4927 generic.go:356] "Generic (PLEG): container finished" podID="427427dd-4d81-40cb-8944-117049c023b4" containerID="8f5e8356bf2c6421be38123e11cd43dabf3a00d9cc275e45895206bd8159f60c" exitCode=0 Oct 30 01:33:08.790185 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:08.781159 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" event={"ID":"427427dd-4d81-40cb-8944-117049c023b4","Type":"ContainerDied","Data":"8f5e8356bf2c6421be38123e11cd43dabf3a00d9cc275e45895206bd8159f60c"} Oct 30 01:33:08.794455 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:08.794301 4927 generic.go:356] "Generic (PLEG): container finished" podID="f67b9aaf-658e-4715-8f99-7f1b45c95833" containerID="b2177326aa33f255d1606284a7a1f043ab699fd063e843bea41c156b9e1ac705" exitCode=0 Oct 30 01:33:08.794455 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:08.794397 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" event={"ID":"f67b9aaf-658e-4715-8f99-7f1b45c95833","Type":"ContainerDied","Data":"b2177326aa33f255d1606284a7a1f043ab699fd063e843bea41c156b9e1ac705"} Oct 30 01:33:09.430978 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.418361 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:33:09.636595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.634010 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67b9aaf-658e-4715-8f99-7f1b45c95833-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67b9aaf-658e-4715-8f99-7f1b45c95833-plugins-config-volume\") pod \"f67b9aaf-658e-4715-8f99-7f1b45c95833\" (UID: \"f67b9aaf-658e-4715-8f99-7f1b45c95833\") " Oct 30 01:33:09.636595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.634466 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67b9aaf-658e-4715-8f99-7f1b45c95833-plugins-config-volume" pod "f67b9aaf-658e-4715-8f99-7f1b45c95833" (UID: "f67b9aaf-658e-4715-8f99-7f1b45c95833"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:09.636595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.634654 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67b9aaf-658e-4715-8f99-7f1b45c95833-kube-api-access-d2vtf\" (UniqueName: \"kubernetes.io/projected/f67b9aaf-658e-4715-8f99-7f1b45c95833-kube-api-access-d2vtf\") pod \"f67b9aaf-658e-4715-8f99-7f1b45c95833\" (UID: \"f67b9aaf-658e-4715-8f99-7f1b45c95833\") " Oct 30 01:33:09.636595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.635395 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67b9aaf-658e-4715-8f99-7f1b45c95833-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:33:09.650128 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.649874 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67b9aaf-658e-4715-8f99-7f1b45c95833-kube-api-access-d2vtf" pod "f67b9aaf-658e-4715-8f99-7f1b45c95833" (UID: "f67b9aaf-658e-4715-8f99-7f1b45c95833"). InnerVolumeSpecName "kube-api-access-d2vtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:09.692428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.692086 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:33:09.737141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.736400 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427427dd-4d81-40cb-8944-117049c023b4-kube-api-access-w7st2\" (UniqueName: \"kubernetes.io/projected/427427dd-4d81-40cb-8944-117049c023b4-kube-api-access-w7st2\") pod \"427427dd-4d81-40cb-8944-117049c023b4\" (UID: \"427427dd-4d81-40cb-8944-117049c023b4\") " Oct 30 01:33:09.737141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.736607 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427427dd-4d81-40cb-8944-117049c023b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427427dd-4d81-40cb-8944-117049c023b4-plugins-config-volume\") pod \"427427dd-4d81-40cb-8944-117049c023b4\" (UID: \"427427dd-4d81-40cb-8944-117049c023b4\") " Oct 30 01:33:09.737141 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.736877 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vtf\" (UniqueName: \"kubernetes.io/projected/f67b9aaf-658e-4715-8f99-7f1b45c95833-kube-api-access-d2vtf\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:33:09.737968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.737621 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427427dd-4d81-40cb-8944-117049c023b4-plugins-config-volume" pod "427427dd-4d81-40cb-8944-117049c023b4" (UID: "427427dd-4d81-40cb-8944-117049c023b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:09.763613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.755821 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427427dd-4d81-40cb-8944-117049c023b4-kube-api-access-w7st2" pod "427427dd-4d81-40cb-8944-117049c023b4" (UID: "427427dd-4d81-40cb-8944-117049c023b4"). InnerVolumeSpecName "kube-api-access-w7st2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:09.802438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.802396 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" Oct 30 01:33:09.802438 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.802413 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt" event={"ID":"f67b9aaf-658e-4715-8f99-7f1b45c95833","Type":"ContainerDied","Data":"e6537bd69e6b2631d72e24e152008555ae02670a9a7ce41fc5ab0c8a5f2b8865"} Oct 30 01:33:09.803050 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.802483 4927 scope.go:122] "RemoveContainer" containerID="b2177326aa33f255d1606284a7a1f043ab699fd063e843bea41c156b9e1ac705" Oct 30 01:33:09.805639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.804279 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" event={"ID":"427427dd-4d81-40cb-8944-117049c023b4","Type":"ContainerDied","Data":"451ec820390c082b6d94706a61daae45718fc54f01f32d8652d7944efb34cb24"} Oct 30 01:33:09.805639 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.804352 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb" Oct 30 01:33:09.814946 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.814931 4927 scope.go:122] "RemoveContainer" containerID="8f5e8356bf2c6421be38123e11cd43dabf3a00d9cc275e45895206bd8159f60c" Oct 30 01:33:09.837836 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.837809 4927 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427427dd-4d81-40cb-8944-117049c023b4-plugins-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:33:09.837836 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.837837 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7st2\" (UniqueName: \"kubernetes.io/projected/427427dd-4d81-40cb-8944-117049c023b4-kube-api-access-w7st2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:33:09.852592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.852524 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb"] Oct 30 01:33:09.855051 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.855024 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ff1bf5-5b45875mvb"] Oct 30 01:33:09.887182 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.887096 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt"] Oct 30 01:33:09.891896 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:09.891864 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ff1bf5-664d69d97fj7nt"] Oct 30 01:33:11.570669 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:11.570634 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427427dd-4d81-40cb-8944-117049c023b4" path="/var/lib/kubelet/pods/427427dd-4d81-40cb-8944-117049c023b4/volumes" Oct 30 01:33:11.571111 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:33:11.571095 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67b9aaf-658e-4715-8f99-7f1b45c95833" path="/var/lib/kubelet/pods/f67b9aaf-658e-4715-8f99-7f1b45c95833/volumes" Oct 30 01:33:34.442328 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:33:34.442244 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:34:04.441363 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:34:04.441318 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:34:20.187488 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.187451 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92"] Oct 30 01:34:20.188189 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.188166 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" Oct 30 01:34:20.188285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.188191 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" Oct 30 01:34:20.188285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.188213 4927 cpu_manager.go:414] "containerMap: removing container" podUID="427427dd-4d81-40cb-8944-117049c023b4" containerName="payload-pre-processing" Oct 30 01:34:20.188285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.188220 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427427dd-4d81-40cb-8944-117049c023b4" containerName="payload-pre-processing" Oct 30 01:34:20.188285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.188235 4927 cpu_manager.go:414] "containerMap: removing container" podUID="f67b9aaf-658e-4715-8f99-7f1b45c95833" containerName="payload-processing" Oct 30 01:34:20.188285 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.188243 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67b9aaf-658e-4715-8f99-7f1b45c95833" containerName="payload-processing" Oct 30 01:34:20.188554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.188349 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" Oct 30 01:34:20.188554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.188363 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="427427dd-4d81-40cb-8944-117049c023b4" containerName="payload-pre-processing" Oct 30 01:34:20.188554 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.188375 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67b9aaf-658e-4715-8f99-7f1b45c95833" containerName="payload-processing" Oct 30 01:34:20.191819 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.191796 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:20.220698 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.220668 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-87vj2\"" Oct 30 01:34:20.224811 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.224784 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92"] Oct 30 01:34:20.359047 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.359010 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/933e528c-7ad8-48d2-899e-29cee218b15e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qbq92\" (UID: \"933e528c-7ad8-48d2-899e-29cee218b15e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:20.359235 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.359057 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbvk\" (UniqueName: \"kubernetes.io/projected/933e528c-7ad8-48d2-899e-29cee218b15e-kube-api-access-nhbvk\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qbq92\" (UID: \"933e528c-7ad8-48d2-899e-29cee218b15e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:20.460174 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.460091 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/933e528c-7ad8-48d2-899e-29cee218b15e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qbq92\" (UID: \"933e528c-7ad8-48d2-899e-29cee218b15e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:20.460174 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.460144 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbvk\" (UniqueName: \"kubernetes.io/projected/933e528c-7ad8-48d2-899e-29cee218b15e-kube-api-access-nhbvk\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qbq92\" (UID: \"933e528c-7ad8-48d2-899e-29cee218b15e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:20.460583 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.460550 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/933e528c-7ad8-48d2-899e-29cee218b15e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qbq92\" (UID: \"933e528c-7ad8-48d2-899e-29cee218b15e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:20.488165 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.488138 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbvk\" (UniqueName: \"kubernetes.io/projected/933e528c-7ad8-48d2-899e-29cee218b15e-kube-api-access-nhbvk\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qbq92\" (UID: \"933e528c-7ad8-48d2-899e-29cee218b15e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:20.505094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.505067 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:20.839848 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:20.839755 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92"] Oct 30 01:34:20.855736 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:34:20.855704 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933e528c_7ad8_48d2_899e_29cee218b15e.slice/crio-24fcfcff59d5b5a1fc9eead5f3b9daf57677090807c3c77e85c0d73fd9c69959 WatchSource:0}: Error finding container 24fcfcff59d5b5a1fc9eead5f3b9daf57677090807c3c77e85c0d73fd9c69959: Status 404 returned error can't find the container with id 24fcfcff59d5b5a1fc9eead5f3b9daf57677090807c3c77e85c0d73fd9c69959 Oct 30 01:34:21.195762 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:21.195722 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" event={"ID":"933e528c-7ad8-48d2-899e-29cee218b15e","Type":"ContainerStarted","Data":"7ec91ae5a8a16d4179a5be9130adbde1d1c327b300b4127a8cd158724f0ab682"} Oct 30 01:34:21.196297 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:21.195783 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" event={"ID":"933e528c-7ad8-48d2-899e-29cee218b15e","Type":"ContainerStarted","Data":"24fcfcff59d5b5a1fc9eead5f3b9daf57677090807c3c77e85c0d73fd9c69959"} Oct 30 01:34:21.196297 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:21.195819 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:21.252044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:21.251983 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" podStartSLOduration=1.251966761 podStartE2EDuration="1.251966761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:21.249334641 +0000 UTC m=+3599.821260862" watchObservedRunningTime="2026-10-30 01:34:21.251966761 +0000 UTC m=+3599.823892982" Oct 30 01:34:22.672501 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:22.672473 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:34:22.701450 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:22.701418 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:34:32.205038 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:34:32.205005 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qbq92" Oct 30 01:34:34.442590 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:34:34.442535 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:35:04.442116 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:35:04.442024 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:35:34.442876 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:35:34.442832 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:36:04.441986 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:36:04.441930 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:36:31.723742 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:36:31.723693 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:36:31.724405 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:36:31.724366 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb" podUID="fd38e17a-6d35-4f06-8ac3-59f578780fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:36:34.442019 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:36:34.441928 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:37:04.507595 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:37:04.506784 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:37:34.443989 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:37:34.443943 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:38:02.322414 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.322377 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-75b4g"] Oct 30 01:38:02.322874 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.322847 4927 cpu_manager.go:414] "containerMap: removing container" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" Oct 30 01:38:02.322874 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.322858 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" Oct 30 01:38:02.322968 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.322958 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e704866-a137-45cd-9aed-3b76635e1409" containerName="istio-proxy" Oct 30 01:38:02.326058 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.326034 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.356996 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.356958 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-75b4g"] Oct 30 01:38:02.442858 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.442763 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93923b89-ce9f-466d-a785-90e92bd695ba-maas-api-tls\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.443044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.442913 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93923b89-ce9f-466d-a785-90e92bd695ba-openshift-service-ca\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.443044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.442993 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93923b89-ce9f-466d-a785-90e92bd695ba-metrics-certs\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.443044 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.443013 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btt2g\" (UniqueName: \"kubernetes.io/projected/93923b89-ce9f-466d-a785-90e92bd695ba-kube-api-access-btt2g\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.543789 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.543744 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93923b89-ce9f-466d-a785-90e92bd695ba-metrics-certs\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.543789 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.543793 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btt2g\" (UniqueName: \"kubernetes.io/projected/93923b89-ce9f-466d-a785-90e92bd695ba-kube-api-access-btt2g\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.544085 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.543855 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93923b89-ce9f-466d-a785-90e92bd695ba-maas-api-tls\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.544085 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.543916 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93923b89-ce9f-466d-a785-90e92bd695ba-openshift-service-ca\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.544628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.544595 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93923b89-ce9f-466d-a785-90e92bd695ba-openshift-service-ca\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.546400 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.546360 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93923b89-ce9f-466d-a785-90e92bd695ba-maas-api-tls\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.546495 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.546461 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93923b89-ce9f-466d-a785-90e92bd695ba-metrics-certs\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.566564 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.566537 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btt2g\" (UniqueName: \"kubernetes.io/projected/93923b89-ce9f-466d-a785-90e92bd695ba-kube-api-access-btt2g\") pod \"maas-api-595cc648c-75b4g\" (UID: \"93923b89-ce9f-466d-a785-90e92bd695ba\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.640079 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.640033 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:02.955122 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.955093 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-75b4g"] Oct 30 01:38:02.957720 ip-10-0-139-66 kubenswrapper[4927]: W1030 01:38:02.957683 4927 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93923b89_ce9f_466d_a785_90e92bd695ba.slice/crio-dc564901bd49207339a19030e85a23294d20545e79b755f816254ea21e24dc8c WatchSource:0}: Error finding container dc564901bd49207339a19030e85a23294d20545e79b755f816254ea21e24dc8c: Status 404 returned error can't find the container with id dc564901bd49207339a19030e85a23294d20545e79b755f816254ea21e24dc8c Oct 30 01:38:02.960458 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:02.959431 4927 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:38:03.936799 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:03.936745 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" event={"ID":"93923b89-ce9f-466d-a785-90e92bd695ba","Type":"ContainerStarted","Data":"43fdaeb200be4a69509c68a435186895ed422e7c7c3408d2542d93223c4afa2b"} Oct 30 01:38:03.937349 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:03.936822 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" event={"ID":"93923b89-ce9f-466d-a785-90e92bd695ba","Type":"ContainerStarted","Data":"dc564901bd49207339a19030e85a23294d20545e79b755f816254ea21e24dc8c"} Oct 30 01:38:03.937592 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:03.937552 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:03.980023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:03.979954 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" podStartSLOduration=1.531835531 podStartE2EDuration="1.979934867s" totalImagesPullingTime="448.099336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:03.977242164 +0000 UTC m=+3822.549168387" watchObservedRunningTime="2026-10-30 01:38:03.979934867 +0000 UTC m=+3822.551861090" Oct 30 01:38:04.622027 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:38:04.621984 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:38:10.998595 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:10.997743 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-75b4g" Oct 30 01:38:11.174608 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:11.152415 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf"] Oct 30 01:38:11.176278 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:11.176230 4927 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" podUID="77df2802-ed6e-40ec-9dca-32a123a63a71" containerName="maas-api" containerID="cri-o://2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f" gracePeriod=30 Oct 30 01:38:12.465718 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.465690 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 01:38:12.520655 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.520623 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77df2802-ed6e-40ec-9dca-32a123a63a71-kube-api-access-c4z5x\" (UniqueName: \"kubernetes.io/projected/77df2802-ed6e-40ec-9dca-32a123a63a71-kube-api-access-c4z5x\") pod \"77df2802-ed6e-40ec-9dca-32a123a63a71\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " Oct 30 01:38:12.520842 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.520671 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-maas-api-tls\") pod \"77df2802-ed6e-40ec-9dca-32a123a63a71\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " Oct 30 01:38:12.520842 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.520726 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-metrics-certs\") pod \"77df2802-ed6e-40ec-9dca-32a123a63a71\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " Oct 30 01:38:12.522885 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.522825 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-metrics-certs" pod "77df2802-ed6e-40ec-9dca-32a123a63a71" (UID: "77df2802-ed6e-40ec-9dca-32a123a63a71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:12.523031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.522877 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77df2802-ed6e-40ec-9dca-32a123a63a71-kube-api-access-c4z5x" pod "77df2802-ed6e-40ec-9dca-32a123a63a71" (UID: "77df2802-ed6e-40ec-9dca-32a123a63a71"). InnerVolumeSpecName "kube-api-access-c4z5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:12.523031 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.522896 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-maas-api-tls" pod "77df2802-ed6e-40ec-9dca-32a123a63a71" (UID: "77df2802-ed6e-40ec-9dca-32a123a63a71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:12.623760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.621554 4927 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77df2802-ed6e-40ec-9dca-32a123a63a71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77df2802-ed6e-40ec-9dca-32a123a63a71-openshift-service-ca\") pod \"77df2802-ed6e-40ec-9dca-32a123a63a71\" (UID: \"77df2802-ed6e-40ec-9dca-32a123a63a71\") " Oct 30 01:38:12.623760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.622024 4927 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77df2802-ed6e-40ec-9dca-32a123a63a71-openshift-service-ca" pod "77df2802-ed6e-40ec-9dca-32a123a63a71" (UID: "77df2802-ed6e-40ec-9dca-32a123a63a71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:12.623760 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.623737 4927 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4z5x\" (UniqueName: \"kubernetes.io/projected/77df2802-ed6e-40ec-9dca-32a123a63a71-kube-api-access-c4z5x\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:38:12.624075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.623762 4927 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-maas-api-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:38:12.624075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.623778 4927 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77df2802-ed6e-40ec-9dca-32a123a63a71-metrics-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:38:12.624075 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:12.623790 4927 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77df2802-ed6e-40ec-9dca-32a123a63a71-openshift-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\"" Oct 30 01:38:13.063755 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.063669 4927 generic.go:356] "Generic (PLEG): container finished" podID="77df2802-ed6e-40ec-9dca-32a123a63a71" containerID="2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f" exitCode=0 Oct 30 01:38:13.063755 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.063714 4927 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" Oct 30 01:38:13.063956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.063752 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" event={"ID":"77df2802-ed6e-40ec-9dca-32a123a63a71","Type":"ContainerDied","Data":"2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f"} Oct 30 01:38:13.063956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.063810 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf" event={"ID":"77df2802-ed6e-40ec-9dca-32a123a63a71","Type":"ContainerDied","Data":"f4a00f5a6dd10f61a466c49c5a357ebc14853bd75fa56087eaf6fc243ba5770c"} Oct 30 01:38:13.063956 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.063837 4927 scope.go:122] "RemoveContainer" containerID="2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f" Oct 30 01:38:13.074498 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.074479 4927 scope.go:122] "RemoveContainer" containerID="2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f" Oct 30 01:38:13.074806 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:38:13.074785 4927 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f\": container with ID starting with 2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f not found: ID does not exist" containerID="2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f" Oct 30 01:38:13.074868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.074817 4927 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f"} err="failed to get container status \"2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f\": rpc error: code = NotFound desc = could not find container \"2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f\": container with ID starting with 2fe64fc44394b6743faf2a64229ac407b43f656746b4316c8f332007795e730f not found: ID does not exist" Oct 30 01:38:13.119618 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.119561 4927 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf"] Oct 30 01:38:13.130471 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.130443 4927 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-m5jgf"] Oct 30 01:38:13.570354 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:13.570315 4927 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77df2802-ed6e-40ec-9dca-32a123a63a71" path="/var/lib/kubelet/pods/77df2802-ed6e-40ec-9dca-32a123a63a71/volumes" Oct 30 01:38:34.442059 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:38:34.442022 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:38:54.780054 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:54.779884 4927 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:38:54.780868 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:38:54.780824 4927 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj" podUID="4e5a86cc-5600-4e82-adfe-a60c090ac69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:39:03.398504 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:03.398465 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-4r98t_371cb075-558e-48f5-a6a3-5f2c5da8196e/manager/0.log" Oct 30 01:39:03.919087 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:03.919049 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f6db8855d-587kv_93c060f9-2b8d-4c98-a339-cde854bb5784/manager/0.log" Oct 30 01:39:04.442363 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:39:04.442323 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:39:05.284155 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:05.284123 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-rmwfm_859beab1-e81b-4ff7-bda3-deb4f372ba44/copy-manifests/0.log" Oct 30 01:39:05.318022 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:05.317996 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-rmwfm_859beab1-e81b-4ff7-bda3-deb4f372ba44/manager/0.log" Oct 30 01:39:05.525442 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:05.525410 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-wx4nc_1a5359b7-b845-4451-a24c-b62c4dac56df/proxy/0.log" Oct 30 01:39:06.280761 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:06.280728 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-75b4g_93923b89-ce9f-466d-a785-90e92bd695ba/maas-api/0.log" Oct 30 01:39:08.310459 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:08.310423 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-bcxfx_e45d6d94-9a44-42c8-a646-210e06552ffa/postgres/0.log" Oct 30 01:39:10.217503 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:10.217469 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-lw8mc_866dff6e-c8f0-4d93-97ee-f6fb9dacdb46/manager/0.log" Oct 30 01:39:10.589605 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:10.589507 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-qbq92_933e528c-7ad8-48d2-899e-29cee218b15e/manager/0.log" Oct 30 01:39:10.996985 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:10.996952 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-wjgtm_52eb857f-d1e0-41a2-95bc-08c5c9d992fb/manager/0.log" Oct 30 01:39:11.742983 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:11.742951 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d9gtglk_3316a59b-b981-4db9-a65e-e615e09625fb/istio-proxy/0.log" Oct 30 01:39:11.923991 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:11.923959 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r4tcs_2e41cfc5-2081-4813-9ac5-a97b47554e53/discovery/0.log" Oct 30 01:39:12.297907 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:12.297877 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d5b5b7d7-t2dm4_75b060a4-e781-4a63-b696-98ce6ad20d2f/kube-auth-proxy/0.log" Oct 30 01:39:12.466959 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:12.466930 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-4fglz_8d429408-e1f3-4a7a-8047-26f5861af69d/istio-proxy/0.log" Oct 30 01:39:12.652497 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:12.652453 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb_fd38e17a-6d35-4f06-8ac3-59f578780fe3/istio-proxy/1.log" Oct 30 01:39:13.060662 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:13.060559 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-885k9_52a414ab-cf1f-479f-b762-e231c92450a1/istio-proxy/0.log" Oct 30 01:39:14.031428 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:14.031379 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj_4e5a86cc-5600-4e82-adfe-a60c090ac69e/istio-proxy/0.log" Oct 30 01:39:14.393609 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:14.393553 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8q7vm_1ae9c7c6-32b3-420d-9cfd-6fc7746096d5/payload-pre-processing/0.log" Oct 30 01:39:14.755694 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:14.755616 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-696c77d7b7-f5pjl_ded47351-6c07-4d2d-a5bc-20dbe69a25ab/router/0.log" Oct 30 01:39:15.220068 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:15.220034 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb_fd38e17a-6d35-4f06-8ac3-59f578780fe3/istio-proxy/0.log" Oct 30 01:39:15.723786 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:15.723748 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk_c19cce81-4a48-4b54-b2ac-29ef57f9395c/main/0.log" Oct 30 01:39:15.898684 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:15.898654 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-fmfpv_6eff56d9-2a99-4b86-9ba0-4bb228804526/main/0.log" Oct 30 01:39:16.249590 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:16.249546 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-k6ckw_857af215-3487-4286-aa40-a9e8b01942a8/main/0.log" Oct 30 01:39:16.805883 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:16.805841 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4_1aedaf3b-4fbd-485d-bf58-319230ad826a/main/0.log" Oct 30 01:39:22.727917 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:22.727879 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:39:22.741687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:22.741661 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:39:26.927197 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:26.927160 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-4r98t_371cb075-558e-48f5-a6a3-5f2c5da8196e/manager/0.log" Oct 30 01:39:27.376837 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:27.376807 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f6db8855d-587kv_93c060f9-2b8d-4c98-a339-cde854bb5784/manager/0.log" Oct 30 01:39:28.677936 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:28.677907 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-rmwfm_859beab1-e81b-4ff7-bda3-deb4f372ba44/copy-manifests/0.log" Oct 30 01:39:28.712063 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:28.712023 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-rmwfm_859beab1-e81b-4ff7-bda3-deb4f372ba44/manager/0.log" Oct 30 01:39:28.920196 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:28.920154 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-wx4nc_1a5359b7-b845-4451-a24c-b62c4dac56df/proxy/0.log" Oct 30 01:39:29.434175 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:29.434144 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-75b4g_93923b89-ce9f-466d-a785-90e92bd695ba/maas-api/0.log" Oct 30 01:39:31.421970 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:31.421935 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-bcxfx_e45d6d94-9a44-42c8-a646-210e06552ffa/postgres/0.log" Oct 30 01:39:32.987620 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:32.987508 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-lw8mc_866dff6e-c8f0-4d93-97ee-f6fb9dacdb46/manager/0.log" Oct 30 01:39:33.369739 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:33.369658 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-qbq92_933e528c-7ad8-48d2-899e-29cee218b15e/manager/0.log" Oct 30 01:39:33.825739 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:33.825710 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-wjgtm_52eb857f-d1e0-41a2-95bc-08c5c9d992fb/manager/0.log" Oct 30 01:39:34.394254 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:34.394225 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d9gtglk_3316a59b-b981-4db9-a65e-e615e09625fb/istio-proxy/0.log" Oct 30 01:39:34.441248 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:39:34.441211 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:39:34.591537 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:34.591501 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r4tcs_2e41cfc5-2081-4813-9ac5-a97b47554e53/discovery/0.log" Oct 30 01:39:34.981412 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:34.981377 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d5b5b7d7-t2dm4_75b060a4-e781-4a63-b696-98ce6ad20d2f/kube-auth-proxy/0.log" Oct 30 01:39:35.177359 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:35.177327 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-4fglz_8d429408-e1f3-4a7a-8047-26f5861af69d/istio-proxy/0.log" Oct 30 01:39:35.365988 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:35.365897 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb_fd38e17a-6d35-4f06-8ac3-59f578780fe3/istio-proxy/1.log" Oct 30 01:39:35.765445 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:35.765413 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-885k9_52a414ab-cf1f-479f-b762-e231c92450a1/istio-proxy/0.log" Oct 30 01:39:36.708239 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:36.708201 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj_4e5a86cc-5600-4e82-adfe-a60c090ac69e/istio-proxy/0.log" Oct 30 01:39:37.097169 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:37.097084 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8q7vm_1ae9c7c6-32b3-420d-9cfd-6fc7746096d5/payload-pre-processing/0.log" Oct 30 01:39:37.474500 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:37.474464 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-696c77d7b7-f5pjl_ded47351-6c07-4d2d-a5bc-20dbe69a25ab/router/0.log" Oct 30 01:39:38.029704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:38.029669 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xqwfk_c19cce81-4a48-4b54-b2ac-29ef57f9395c/main/0.log" Oct 30 01:39:38.231970 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:38.231939 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-fmfpv_6eff56d9-2a99-4b86-9ba0-4bb228804526/main/0.log" Oct 30 01:39:38.620820 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:38.620783 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-k6ckw_857af215-3487-4286-aa40-a9e8b01942a8/main/0.log" Oct 30 01:39:39.176774 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:39:39.176734 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5tkv4_1aedaf3b-4fbd-485d-bf58-319230ad826a/main/0.log" Oct 30 01:40:04.441891 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:40:04.441851 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:40:09.890860 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:09.890830 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f6db8855d-587kv_93c060f9-2b8d-4c98-a339-cde854bb5784/manager/0.log" Oct 30 01:40:10.170713 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:10.170628 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f6db8855d-587kv_93c060f9-2b8d-4c98-a339-cde854bb5784/manager/0.log" Oct 30 01:40:28.806210 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:28.806178 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 01:40:34.441767 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:40:34.441728 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:40:50.347019 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:50.346982 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f6db8855d-587kv_93c060f9-2b8d-4c98-a339-cde854bb5784/manager/0.log" Oct 30 01:40:50.621831 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:50.621751 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f6db8855d-587kv_93c060f9-2b8d-4c98-a339-cde854bb5784/manager/0.log" Oct 30 01:40:56.175899 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:56.175868 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2rkgf_5ad24271-c6fb-4ac0-b2b5-b915db5b61ec/global-pull-secret-syncer/0.log" Oct 30 01:40:56.804778 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:56.804749 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kkhpn_43af9500-17cd-44ee-9625-de42cf5c4c42/connection-checker/0.log" Oct 30 01:40:56.927274 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:56.927247 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-w2hfm_0432d327-4236-4bd4-8f39-d2dfe783e07e/connection-checker/0.log" Oct 30 01:40:57.053415 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:57.053389 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xkfl8_3f53e15a-0cbc-4a61-ae76-9ddc8996cabc/connection-checker/0.log" Oct 30 01:40:57.503699 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:57.503656 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xvxrc_286e9e4c-1a89-4c7c-90b8-222e5f48af2c/konnectivity-agent/0.log" Oct 30 01:40:57.947417 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:40:57.947385 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-66.ec2.internal_07ffc31fbc22d538689cf3a7345b3440/haproxy/0.log" Oct 30 01:41:04.058613 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:04.058521 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-lw8mc_866dff6e-c8f0-4d93-97ee-f6fb9dacdb46/manager/0.log" Oct 30 01:41:04.318973 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:04.318895 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-qbq92_933e528c-7ad8-48d2-899e-29cee218b15e/manager/0.log" Oct 30 01:41:04.442721 ip-10-0-139-66 kubenswrapper[4927]: E1030 01:41:04.442679 4927 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9pkrm" containerName="extractor" Oct 30 01:41:04.622369 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:04.622286 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-wjgtm_52eb857f-d1e0-41a2-95bc-08c5c9d992fb/manager/0.log" Oct 30 01:41:06.639522 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.639483 4927 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj"] Oct 30 01:41:06.640013 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.639997 4927 cpu_manager.go:414] "containerMap: removing container" podUID="77df2802-ed6e-40ec-9dca-32a123a63a71" containerName="maas-api" Oct 30 01:41:06.640064 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.640014 4927 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77df2802-ed6e-40ec-9dca-32a123a63a71" containerName="maas-api" Oct 30 01:41:06.640116 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.640105 4927 memory_manager.go:365] "RemoveStaleState removing state" podUID="77df2802-ed6e-40ec-9dca-32a123a63a71" containerName="maas-api" Oct 30 01:41:06.643413 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.643392 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.668380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.668347 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hpkt\"/\"openshift-service-ca.crt\"" Oct 30 01:41:06.668380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.668357 4927 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hpkt\"/\"kube-root-ca.crt\"" Oct 30 01:41:06.668380 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.668345 4927 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8hpkt\"/\"default-dockercfg-mlw7p\"" Oct 30 01:41:06.671847 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.671823 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj"] Oct 30 01:41:06.723797 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.723759 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnj4h\" (UniqueName: \"kubernetes.io/projected/7d09d40e-c5de-43d3-965b-6491a1e7634b-kube-api-access-pnj4h\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.723958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.723801 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-lib-modules\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.723958 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.723826 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-podres\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.724059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.724006 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-proc\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.724059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.724055 4927 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-sys\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.825619 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.825559 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-proc\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.825796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.825641 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-sys\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.825796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.825669 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnj4h\" (UniqueName: \"kubernetes.io/projected/7d09d40e-c5de-43d3-965b-6491a1e7634b-kube-api-access-pnj4h\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.825796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.825693 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-lib-modules\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.825796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.825732 4927 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-podres\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.825796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.825748 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-sys\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.825796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.825768 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-lib-modules\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.825796 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.825690 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-proc\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.826059 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.825816 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7d09d40e-c5de-43d3-965b-6491a1e7634b-podres\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.849704 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.849679 4927 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnj4h\" (UniqueName: \"kubernetes.io/projected/7d09d40e-c5de-43d3-965b-6491a1e7634b-kube-api-access-pnj4h\") pod \"perf-node-gather-daemonset-7m9wj\" (UID: \"7d09d40e-c5de-43d3-965b-6491a1e7634b\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:06.889339 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.889306 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-7b7wb_62391a80-1181-4e4b-98a2-bcc1c692f460/cluster-monitoring-operator/0.log" Oct 30 01:41:06.956707 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:06.956631 4927 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:07.276392 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:07.276361 4927 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj"] Oct 30 01:41:07.595246 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:07.595149 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g9nnw_0a393f07-726f-49bb-925f-913d3c4c729a/node-exporter/0.log" Oct 30 01:41:07.728094 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:07.728063 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g9nnw_0a393f07-726f-49bb-925f-913d3c4c729a/kube-rbac-proxy/0.log" Oct 30 01:41:07.845023 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:07.844995 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g9nnw_0a393f07-726f-49bb-925f-913d3c4c729a/init-textfile/0.log" Oct 30 01:41:08.113997 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:08.113957 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" event={"ID":"7d09d40e-c5de-43d3-965b-6491a1e7634b","Type":"ContainerStarted","Data":"210f0f09dc582e53392d4688e7311e599282fd7fda7c50da685bee53339384c5"} Oct 30 01:41:08.114196 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:08.114010 4927 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" event={"ID":"7d09d40e-c5de-43d3-965b-6491a1e7634b","Type":"ContainerStarted","Data":"6bdf54282baca7e726c8f78bdfa07070120638d3028e81e6c01253c932201b84"} Oct 30 01:41:08.114196 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:08.114038 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:08.155628 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:08.155562 4927 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" podStartSLOduration=2.155546912 podStartE2EDuration="2.155546912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:08.152776635 +0000 UTC m=+4006.724702857" watchObservedRunningTime="2026-10-30 01:41:08.155546912 +0000 UTC m=+4006.727473180" Oct 30 01:41:12.648622 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:12.648588 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-6d46m_36441a12-4446-4269-b6fd-61d82a00d242/networking-console-plugin/0.log" Oct 30 01:41:13.660687 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:13.660649 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/2.log" Oct 30 01:41:13.664397 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:13.664361 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gbrdh_a2a1f298-60ee-4efe-8cec-88db4036a4e8/console-operator/3.log" Oct 30 01:41:14.130962 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:14.130935 4927 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-7m9wj" Oct 30 01:41:15.940348 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:15.940318 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-2snxj_0203176b-a80e-4521-8bf3-833d5b973164/volume-data-source-validator/0.log" Oct 30 01:41:18.363311 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:18.363268 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gbrwr_59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70/dns/0.log" Oct 30 01:41:18.505236 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:18.505186 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gbrwr_59fcdf40-4ce4-4d7f-ba4f-a8cc8a405c70/kube-rbac-proxy/0.log" Oct 30 01:41:19.414206 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:19.414160 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdz28_8fbd9c3e-2c40-45f6-b808-76dde38075ef/dns-node-resolver/0.log" Oct 30 01:41:19.880291 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:19.880256 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d9gtglk_3316a59b-b981-4db9-a65e-e615e09625fb/istio-proxy/0.log" Oct 30 01:41:20.375668 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:20.375623 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r4tcs_2e41cfc5-2081-4813-9ac5-a97b47554e53/discovery/0.log" Oct 30 01:41:20.481552 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:20.481513 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d5b5b7d7-t2dm4_75b060a4-e781-4a63-b696-98ce6ad20d2f/kube-auth-proxy/0.log" Oct 30 01:41:20.550416 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:20.550379 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-4fglz_8d429408-e1f3-4a7a-8047-26f5861af69d/istio-proxy/0.log" Oct 30 01:41:20.623405 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:20.623355 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb_fd38e17a-6d35-4f06-8ac3-59f578780fe3/istio-proxy/0.log" Oct 30 01:41:20.625048 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:20.625023 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb_fd38e17a-6d35-4f06-8ac3-59f578780fe3/istio-proxy/1.log" Oct 30 01:41:20.739953 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:20.739855 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-885k9_52a414ab-cf1f-479f-b762-e231c92450a1/istio-proxy/0.log" Oct 30 01:41:20.756478 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:20.756454 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-274sk_17afbfde-0e2f-4411-bdb2-bc9b3f5a04f3/node-ca/0.log" Oct 30 01:41:21.137242 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:21.137204 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-sq9nj_4e5a86cc-5600-4e82-adfe-a60c090ac69e/istio-proxy/0.log" Oct 30 01:41:21.273780 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:21.273749 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8q7vm_1ae9c7c6-32b3-420d-9cfd-6fc7746096d5/payload-pre-processing/0.log" Oct 30 01:41:21.415806 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:21.415731 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-696c77d7b7-f5pjl_ded47351-6c07-4d2d-a5bc-20dbe69a25ab/router/0.log" Oct 30 01:41:22.861833 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:22.861804 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d9gtglk_3316a59b-b981-4db9-a65e-e615e09625fb/istio-proxy/0.log" Oct 30 01:41:23.408295 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:23.408258 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r4tcs_2e41cfc5-2081-4813-9ac5-a97b47554e53/discovery/0.log" Oct 30 01:41:24.296156 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:24.296122 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d5b5b7d7-t2dm4_75b060a4-e781-4a63-b696-98ce6ad20d2f/kube-auth-proxy/0.log" Oct 30 01:41:24.418865 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:24.418835 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-4fglz_8d429408-e1f3-4a7a-8047-26f5861af69d/istio-proxy/0.log" Oct 30 01:41:24.554998 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:24.554904 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb_fd38e17a-6d35-4f06-8ac3-59f578780fe3/istio-proxy/0.log" Oct 30 01:41:24.556241 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:24.556219 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-5bzrb_fd38e17a-6d35-4f06-8ac3-59f578780fe3/istio-proxy/1.log" Oct 30 01:41:24.771726 ip-10-0-139-66 kubenswrapper[4927]: I1030 01:41:24.771692 4927 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-885k9_52a414ab-cf1f-479f-b762-e231c92450a1/istio-proxy/0.log"